Each session in this course uses R code for demonstration. All the content is self-contained within a R software package designed for the course and uses notebooks for each session.
+
If you would like to you can follow the course material without installing any software or running any code. However, if you would like to run the code yourself, you will need to follow the instructions below.
+
+
+
Getting R
+
+
R is used as the main programming language. Please install at least version: R-4.2.0.
+
RStudio is a popular graphic user interface (GUI). Its Visual Editor provides the best experience of going through this course. Please make sure you update RStudio to the latest version.
+
+
+
+
Installing additional requirements
+
Before you get started with the course, you will first need to install the following software.
+
+
Installation of the nfidd package
+
To install the packages needed in the course, including the nfidd package that contains data files and helper functions used throughout, you can use the pak package:
Alternatively, if you are familiar with git you can clone the repo.
+
If you prefer, you can also view each session on the website, and copy-paste the code into your own R script. In that case you don’t need to download the material.
+
+
Tip: if you hover over each code chunk on the website you can use a “Copy” button at the top right corner.
+
+
+
+
Interacting with a local copy of the course material
+
A benefit of downloading or cloning all the material is that you can interact with the session files directly.
+
In this course, all content is written using R Notebooks. This means that we can combine text with code and see the output directly. The notebooks are then directly reproduced on the course website (for example, this page).
+
To interact with each session in the course, we recommend opening the RStudio Project file (ueifid.RProj) in the ueifid folder you have just downloaded. Then you can choose to:
+
+
View each session on the website, and copy-paste the code into your own R script.
+
+
Tip: if you hover over each code chunk on the website you can use a “Copy” button at the top right corner.
+
+
Open the R Notebook for each session.
+
+
Each notebook is saved in ueifid/sessions/ as a .qmd file.
+
Execute code with the single green “play” button at the top-right corner of each code chunk (“Run current chunk”). You can also execute code line-by-line using Ctrl/Cmd + Enter.
+
We suggest using “Visual” view for a better experience (top-left of the RStudio pane).
understanding of forecasting as an epidemiological problem
+
understanding of ways to visualise forecasts
+
+
+
+
Evaluating forecasts
+
+
familiarity with metrics for evaluating probabilistic forecasts and their properties
+
ability to score probabilistic forecasts in R
+
+
+
+
Ensemble models
+
+
understanding of predictive ensembles and their properties
+
ability to create a predictive ensemble of forecasts in R
+
understanding the concept of weighted forecast ensembles
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/robots.txt b/robots.txt
new file mode 100644
index 0000000..4ed394c
--- /dev/null
+++ b/robots.txt
@@ -0,0 +1 @@
+Sitemap: https://nfidd.github.io/ueifid/sitemap.xml
diff --git a/search.json b/search.json
new file mode 100644
index 0000000..3c1cc51
--- /dev/null
+++ b/search.json
@@ -0,0 +1,663 @@
+[
+ {
+ "objectID": "sessions/forecast-ensembles.html",
+ "href": "sessions/forecast-ensembles.html",
+ "title": "Forecast ensembles",
+ "section": "",
+ "text": "As we saw in the forecast evaluation session, different modelling approaches have different strength and weaknesses, and it is not clear a priori which one produces the best forecast in any given situation. One way to attempt to draw strength from a diversity of approaches is the creation of so-called forecast ensembles from the forecasts produced by different models.\nIn this session, we’ll build ensembles using forecasts from models of different levels of mechanism vs. statistical complexity. We will then compare the performance of these ensembles to the individual models and to each other. Rather than using the forecast samples we have been using we will instead now use quantile-based forecasts.\n\n\n\n\n\n\nRepresentations of probabilistic forecasts\n\n\n\n\n\nProbabilistic predictions can be described as coming from a probabilistic probability distributions. In general and when using complex models such as the one we discuss in this course, these distributions can not be expressed in a simple analytical formal as we can do if, e.g. talking about common probability distributions such as the normal or gamma distributions. Instead, we typically use a limited number of samples generated from Monte-Carlo methods to represent the predictive distribution. However, this is not the only way to characterise distributions.\nA quantile is the value that corresponds to a given quantile level of a distribution. For example, the median is the 50th quantile of a distribution, meaning that 50% of the values in the distribution are less than the median and 50% are greater. Similarly, the 90th quantile is the value that corresponds to 90% of the distribution being less than this value. If we characterise a predictive distribution by its quantiles, we specify these values at a range of specific quantile levels, e.g. from 5% to 95% in 5% steps.\nDeciding how to represent forecasts depends on many things, for example the method used (and whether it produces samples by default) but also logistic considerations. Many collaborative forecasting projects and so-called forecasting hubs use quantile-based representations of forecasts in the hope to be able to characterise both the centre and tails of the distributions more reliably and with less demand on storage space than a sample-based representation.\n\n\n\n\n\n\nIntroduction to ensembles\n\n\n\n\nThe aim of this session is to introduce the concept of ensembles of forecasts and to evaluate the performance of ensembles of the multiple models.\n\n\n\n\n\n\nSetup\n\n\n\n\n\n\n\nThe source file of this session is located at sessions/forecast-ensembles.qmd.\n\n\n\nIn this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and tidyr packages for data wrangling, ggplot2 library for plotting, the tidybayes package for extracting results of the inference and the scoringutils package for evaluating forecasts. We will also use qrensemble for quantile regression averaging in the weighted ensemble section.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"tidyr\")\nlibrary(\"ggplot2\")\nlibrary(\"scoringutils\")\nlibrary(\"qrensemble\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.\n\n\n\n\n\nWe set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#slides",
+ "href": "sessions/forecast-ensembles.html#slides",
+ "title": "Forecast ensembles",
+ "section": "",
+ "text": "Introduction to ensembles",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#objectives",
+ "href": "sessions/forecast-ensembles.html#objectives",
+ "title": "Forecast ensembles",
+ "section": "",
+ "text": "The aim of this session is to introduce the concept of ensembles of forecasts and to evaluate the performance of ensembles of the multiple models.\n\n\n\n\n\n\nSetup\n\n\n\n\n\n\n\nThe source file of this session is located at sessions/forecast-ensembles.qmd.\n\n\n\nIn this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and tidyr packages for data wrangling, ggplot2 library for plotting, the tidybayes package for extracting results of the inference and the scoringutils package for evaluating forecasts. We will also use qrensemble for quantile regression averaging in the weighted ensemble section.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"tidyr\")\nlibrary(\"ggplot2\")\nlibrary(\"scoringutils\")\nlibrary(\"qrensemble\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.\n\n\n\n\n\nWe set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#source-file",
+ "href": "sessions/forecast-ensembles.html#source-file",
+ "title": "Forecast ensembles",
+ "section": "",
+ "text": "The source file of this session is located at sessions/forecast-ensembles.qmd.",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#libraries-used",
+ "href": "sessions/forecast-ensembles.html#libraries-used",
+ "title": "Forecast ensembles",
+ "section": "",
+ "text": "In this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and tidyr packages for data wrangling, ggplot2 library for plotting, the tidybayes package for extracting results of the inference and the scoringutils package for evaluating forecasts. We will also use qrensemble for quantile regression averaging in the weighted ensemble section.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"tidyr\")\nlibrary(\"ggplot2\")\nlibrary(\"scoringutils\")\nlibrary(\"qrensemble\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#initialisation",
+ "href": "sessions/forecast-ensembles.html#initialisation",
+ "title": "Forecast ensembles",
+ "section": "",
+ "text": "We set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#construction",
+ "href": "sessions/forecast-ensembles.html#construction",
+ "title": "Forecast ensembles",
+ "section": "Construction",
+ "text": "Construction\nWe can calculate the mean ensemble by taking the mean of the forecasts at each quantile level.\n\nmean_ensemble <- quantile_forecasts |>\n as_tibble() |>\n summarise(\n predicted = mean(predicted),\n observed = unique(observed),\n model = \"Mean ensemble\",\n .by = c(target_day, horizon, quantile_level, day)\n )\n\nSimilarly, we can calculate the median ensemble by taking the median of the forecasts at each quantile level.\n\nmedian_ensemble <- quantile_forecasts |>\n as_tibble() |>\n summarise(\n predicted = median(predicted),\n observed = unique(observed),\n model = \"Median ensemble\",\n .by = c(target_day, horizon, quantile_level, day)\n )\n\nWe combine the ensembles into a single data frame along with the individual forecasts in order to make visualisation easier.\n\nsimple_ensembles <- bind_rows(\n mean_ensemble,\n median_ensemble,\n quantile_forecasts\n)",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#visualisation",
+ "href": "sessions/forecast-ensembles.html#visualisation",
+ "title": "Forecast ensembles",
+ "section": "Visualisation",
+ "text": "Visualisation\nHow do these ensembles visually differ from the individual models? Lets start by plotting a single forecast from each model and comparing them.\n\nplot_ensembles <- function(data, obs_data) {\n data |>\n pivot_wider(names_from = quantile_level, values_from = predicted) |>\n ggplot(aes(x = day)) +\n geom_ribbon(\n aes(\n ymin = .data[[\"0.05\"]], ymax = .data[[\"0.95\"]], fill = model,\n group = target_day\n ),\n alpha = 0.2\n ) +\n geom_ribbon(\n aes(\n ymin = .data[[\"0.25\"]], ymax = .data[[\"0.75\"]], fill = model,\n group = target_day\n ),\n alpha = 0.5\n ) +\n geom_point(\n data = obs_data,\n aes(x = day, y = onsets), color = \"black\"\n ) +\n scale_color_binned(type = \"viridis\") +\n facet_wrap(~model) +\n theme(legend.position = \"none\")\n}\n\nplot_single_forecasts <- simple_ensembles |>\n filter(target_day == 57) |>\n plot_ensembles(onset_df |> filter(day >= 57, day <= 57 + 14))\n\nplot_single_forecasts\n\n\n\n\n\n\n\n\nAgain we can get a different perspective by plotting the forecasts on the log scale.\n\nplot_single_forecasts +\n scale_y_log10()\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nHow do these ensembles compare to the individual models? How do they differ from each other?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nHow do these ensembles compare to the individual models?\n\nBoth of the simple ensembles appear to be less variable than the statistical models but are more variable than the mechanistic model.\nBoth ensembles are more like the statistical models than the mechanistic model.\n\nHow do they differ from each other?\n\nThe mean ensemble has slightly tighter uncertainty bounds than the median ensemble.\n\n\n\n\nNow lets plot a range of forecasts from each model and ensemble.\n\nplot_multiple_forecasts <- simple_ensembles |>\n plot_ensembles(onset_df |> filter(day >= 21)) +\n lims(y = c(0, 400))\n\nplot_multiple_forecasts\n\n\n\n\n\n\n\n\nAgain we can get a different perspective by plotting the forecasts on the log scale.\n\nplot_multiple_forecasts +\n scale_y_log10()\n\nScale for y is already present.\nAdding another scale for y, which will replace the existing scale.\n\n\nWarning in scale_y_log10(): log-10 transformation introduced infinite values.\nlog-10 transformation introduced infinite values.\nlog-10 transformation introduced infinite values.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nHow do these ensembles compare to the individual models?\nHow do they differ from each other?\nAre there any differences across forecast dates?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nHow do these ensembles compare to the individual models?\n\nAs before, the ensembles appear to be less variable than the statistical models but more variable than the mechanistic model.\n\nHow do they differ from each other?\n\nThe mean ensemble has marginally tighter uncertainty bounds than the median ensemble as for the single forecast.\n\nAre there any differences across forecast dates?\n\nThe mean ensemble appears to be more variable across forecast dates than the median ensemble with this being more pronounced after the peak of the outbreak.",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#evaluation",
+ "href": "sessions/forecast-ensembles.html#evaluation",
+ "title": "Forecast ensembles",
+ "section": "Evaluation",
+ "text": "Evaluation\nAs in the forecast evaluation session, we can evaluate the accuracy of the ensembles using the {scoringutils} package and in particular the score() function.\n\nensemble_scores <- simple_ensembles |>\n as_forecast_quantile(forecast_unit = c(\"target_day\", \"horizon\", \"model\")) |>\n score()\n\n\n\n\n\n\n\nNote\n\n\n\nThe weighted interval score (WIS) is a proper scoring rule for quantile forecasts that approximates the Continuous Ranked Probability Score (CRPS) by considering a weighted sum of multiple prediction intervals. As the number of intervals increases, the WIS converges to the CRPS, combining sharpness and penalties for over- and under-prediction.\nWe see it here as we are scoring quantiles and not samples hence we cannot use CRPS as we did before.\n\n\nAgain we start with a high level overview of the scores by model.\n\nensemble_scores |>\n summarise_scores(by = c(\"model\"))\n\n model wis overprediction underprediction dispersion\n <char> <num> <num> <num> <num>\n1: Mean ensemble 8.250097 4.271101 0.9264881 3.052507\n2: Median ensemble 10.197812 5.954107 0.9897321 3.253973\n3: Random walk 12.086955 7.568929 0.7236607 3.794366\n4: More statistical 10.717272 5.710446 1.3504464 3.656379\n5: More mechanistic 5.453563 1.614821 2.1319643 1.706777\n bias interval_coverage_50 interval_coverage_90 ae_median\n <num> <num> <num> <num>\n1: 0.200000000 0.5133929 0.8750000 12.498512\n2: 0.158928571 0.5267857 0.8794643 15.484375\n3: 0.227678571 0.4955357 0.8616071 18.209821\n4: -0.004464286 0.5357143 0.8616071 15.959821\n5: 0.214285714 0.4732143 0.7857143 8.473214\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nWhat do you think the scores are telling you? Which model do you think is best? What other scoring breakdowns might you want to look at?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nWhat do you think the scores are telling you? Which model do you think is best?\n\nThe mean ensemble appears to be the best performing ensemble model overall.\nHowever, the more mechanistic model appears to be the best performing model overall.\n\nWhat other scoring breakdowns might you want to look at?\n\nThere might be variation over forecast dates or horizons between the different ensemble methods",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#construction-1",
+ "href": "sessions/forecast-ensembles.html#construction-1",
+ "title": "Forecast ensembles",
+ "section": "Construction",
+ "text": "Construction\nAs we just saw, the random walk model (our original baseline model) is performing poorly in comparison to the other models. We can remove this model from the ensemble and see if this improves the performance of the ensemble.\n\n\n\n\n\n\nWarning\n\n\n\nHere we are technically cheating a little as we are using the test data to help select the models to include in the ensemble. In the real world you would not do this as you would not have access to the test data and so this is an idealised scenario.\n\n\n\nfiltered_models <- quantile_forecasts |>\n filter(model != \"Random walk\")\n\nWe then need to recalculate the ensembles. First the mean ensemble,\n\nfiltered_mean_ensembles <- filtered_models |>\n as_tibble() |>\n summarise(\n predicted = mean(predicted),\n observed = unique(observed),\n model = \"Mean filtered ensemble\",\n .by = c(target_day, horizon, quantile_level, day)\n )\n\nand then the median ensemble.\n\nfiltered_median_ensembles <- filtered_models |>\n as_tibble() |>\n summarise(\n predicted = median(predicted),\n observed = unique(observed),\n model = \"Median filtered ensemble\",\n .by = c(target_day, horizon, quantile_level, day)\n )\n\nWe combine these new ensembles with our previous ensembles in order to make visualisation easier.\n\nfiltered_ensembles <- bind_rows(\n filtered_mean_ensembles,\n filtered_median_ensembles,\n simple_ensembles\n)",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#visualisation-1",
+ "href": "sessions/forecast-ensembles.html#visualisation-1",
+ "title": "Forecast ensembles",
+ "section": "Visualisation",
+ "text": "Visualisation\nAs for the simple ensembles, we can plot a single forecast from each model and ensemble.\n\nfiltered_ensembles |>\n filter(target_day == 57) |>\n plot_ensembles(onset_df |> filter(day >= 57, day <= 57 + 14))\n\n\n\n\n\n\n\n\nand on the log scale.\n\nfiltered_ensembles |>\n filter(target_day == 57) |>\n plot_ensembles(onset_df |> filter(day >= 57, day <= 57 + 14)) +\n scale_y_log10()\n\n\n\n\n\n\n\n\nTo get an overview we also plot a range of forecasts from each model and ensemble.\n\nplot_multiple_filtered_forecasts <- filtered_ensembles |>\n plot_ensembles(onset_df |> filter(day >= 21)) +\n lims(y = c(0, 400))\nplot_multiple_filtered_forecasts\n\n\n\n\n\n\n\n\nand on the log scale.\n\nplot_multiple_filtered_forecasts +\n scale_y_log10()\n\nScale for y is already present.\nAdding another scale for y, which will replace the existing scale.\n\n\nWarning in scale_y_log10(): log-10 transformation introduced infinite values.\nlog-10 transformation introduced infinite values.\nlog-10 transformation introduced infinite values.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nHow do the filtered ensembles compare to the simple ensembles? Which do you think is best?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nHow do the filtered ensembles compare to the simple ensembles?\n\nThe filtered ensembles appear to be less variable than the simple ensembles.\nThe filtered ensembles appear to be more like the mechanistic model than the simple ensembles.\n\nWhich do you think is best?\n\nVisually, the filtered ensembles appear very similar. This makes sense given we know there are only two models left in the ensemble.",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#evaluation-1",
+ "href": "sessions/forecast-ensembles.html#evaluation-1",
+ "title": "Forecast ensembles",
+ "section": "Evaluation",
+ "text": "Evaluation\nLet us score the filtered ensembles.\n\nfiltered_ensemble_scores <- filtered_ensembles |>\n as_forecast_quantile(\n forecast_unit = c(\n \"target_day\", \"horizon\", \"model\"\n )\n ) |>\n score()\n\nAgain we can get a high level overview of the scores by model.\n\nfiltered_ensemble_scores |>\n summarise_scores(by = c(\"model\"))\n\n model wis overprediction underprediction dispersion\n <char> <num> <num> <num> <num>\n1: Mean filtered ensemble 6.754926 2.891027 1.1823214 2.681578\n2: Median filtered ensemble 6.754926 2.891027 1.1823214 2.681578\n3: Mean ensemble 8.250097 4.271101 0.9264881 3.052507\n4: Median ensemble 10.197812 5.954107 0.9897321 3.253973\n5: Random walk 12.086955 7.568929 0.7236607 3.794366\n6: More statistical 10.717272 5.710446 1.3504464 3.656379\n7: More mechanistic 5.453563 1.614821 2.1319643 1.706777\n bias interval_coverage_50 interval_coverage_90 ae_median\n <num> <num> <num> <num>\n1: 0.152232143 0.5312500 0.8794643 10.337054\n2: 0.152232143 0.5312500 0.8794643 10.337054\n3: 0.200000000 0.5133929 0.8750000 12.498512\n4: 0.158928571 0.5267857 0.8794643 15.484375\n5: 0.227678571 0.4955357 0.8616071 18.209821\n6: -0.004464286 0.5357143 0.8616071 15.959821\n7: 0.214285714 0.4732143 0.7857143 8.473214\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nHow do the filtered ensembles compare to the simple ensembles?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nHow do the filtered ensembles compare to the simple ensembles?\n\nThe filtered ensembles appear to be more accurate than the simple ensembles.\nAs you would expect they are an average of the more mechanistic model and the more statistical model.\nAs there are only two models in the ensemble, the median and mean ensembles are identical.\nFor the first time there are features of the ensemble that outperform the more mechanistic model though it remains the best performing model overall.",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#construction-2",
+ "href": "sessions/forecast-ensembles.html#construction-2",
+ "title": "Forecast ensembles",
+ "section": "Construction",
+ "text": "Construction\n\nInverse WIS weighting\nIn order to perform inverse WIS weighting we first need to calculate the WIS for each model. We already have this from the previous evaluation so we can reuse this.\n\nweights_per_model <- ensemble_scores |>\n dplyr::filter(\n model %in% c(\"More mechanistic\", \"More statistical\", \"Random walk\")\n ) |>\n summarise_scores(by = c(\"model\", \"target_day\")) |>\n select(model, target_day, wis) |>\n mutate(inv_wis = 1 / wis) |>\n mutate(\n inv_wis_total_by_date = sum(inv_wis), .by = target_day\n ) |>\n mutate(weights = inv_wis / inv_wis_total_by_date)\n\nweights_per_model |>\n select(model, target_day, weights) |>\n pivot_wider(names_from = model, values_from = weights)\n\n# A tibble: 16 × 4\n target_day `Random walk` `More statistical` `More mechanistic`\n <dbl> <dbl> <dbl> <dbl>\n 1 22 0.323 0.379 0.298\n 2 29 0.408 0.346 0.247\n 3 36 0.378 0.287 0.335\n 4 43 0.324 0.267 0.409\n 5 50 0.297 0.226 0.477\n 6 57 0.290 0.248 0.462\n 7 64 0.243 0.264 0.494\n 8 71 0.418 0.326 0.256\n 9 78 0.233 0.334 0.432\n10 85 0.0652 0.0748 0.860\n11 92 0.215 0.300 0.484\n12 99 0.205 0.304 0.490\n13 106 0.151 0.177 0.672\n14 113 0.324 0.470 0.206\n15 120 0.440 0.387 0.173\n16 127 0.319 0.260 0.421\n\n\nNow lets apply the weights to the forecast models. As we can only use information that was available at the time of the forecast to perform the weighting, we use weights from two weeks prior to the forecast date to inform each ensemble.\n\ninverse_wis_ensemble <- quantile_forecasts |>\n as_tibble() |>\n left_join(\n weights_per_model |>\n mutate(target_day = target_day + 14),\n by = c(\"model\", \"target_day\")\n ) |>\n # assign equal weights if no weights are available\n mutate(weights = ifelse(is.na(weights), 1/3, weights)) |>\n summarise(\n predicted = sum(predicted * weights),\n observed = unique(observed),\n model = \"Inverse WIS ensemble\",\n .by = c(target_day, horizon, quantile_level, day)\n )\n\n\n\nQuantile regression averaging\nWe futher to perform quantile regression averaging (QRA) for each forecast date. Again we need to consider how many previous forecasts we wish to use to inform each ensemble forecast. Here we decide to use up to 3 weeks of previous forecasts to inform each QRA ensemble.\n\nforecast_dates <- quantile_forecasts |>\n as_tibble() |>\n pull(target_day) |>\n unique()\n\nqra_by_forecast <- function(\n quantile_forecasts,\n forecast_dates,\n group = c(\"target_end_date\"), \n ...\n) {\n lapply(forecast_dates, \\(x) {\n quantile_forecasts |>\n mutate(target_end_date = x) |>\n dplyr::filter(target_day <= x) |>\n dplyr::filter(target_day >= x - (3 * 7 + 1)) |>\n dplyr::filter(target_day == x | day <= x) |>\n qra(\n group = group,\n target = c(target_day = x),\n ...\n )\n })\n}\n\nqra_ensembles_obj <- qra_by_forecast(\n quantile_forecasts,\n forecast_dates[-1],\n group = c(\"target_end_date\")\n)\n\nqra_weights <- seq_along(qra_ensembles_obj) |>\n lapply(\\(i) attr(qra_ensembles_obj[[i]], \"weights\") |>\n mutate(target_day = forecast_dates[i + 1])\n ) |>\n bind_rows() |>\n dplyr::filter(quantile == 0.5)\n\nqra_ensembles <- qra_ensembles_obj |>\n bind_rows()\n\nInstead of creating a single optimised ensemble and using this for all forecast horizons we might also want to consider a separate optimised QRA ensemble for each forecast horizon, reflecting that models might perform differently depending on how far ahead a forecast is produced. We can do this using qra() with the group argument.\n\nqra_ensembles_by_horizon <- qra_by_forecast(\n quantile_forecasts,\n forecast_dates[-c(1:2)],\n group = c(\"horizon\", \"target_end_date\"),\n model = \"QRA by horizon\"\n)\n\nqra_weights_by_horizon <- seq_along(qra_ensembles_by_horizon) |>\n lapply(\\(i) attr(qra_ensembles_by_horizon[[i]], \"weights\") |>\n mutate(target_day = forecast_dates[i + 2])\n ) |>\n bind_rows()\n\n\n\nCombine ensembles\n\nweighted_ensembles <- bind_rows(\n inverse_wis_ensemble,\n qra_ensembles,\n qra_ensembles_by_horizon,\n filtered_ensembles\n) |>\n # remove the repeated filtered ensemble\n filter(model != \"Mean filtered ensemble\")",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#visualisation-2",
+ "href": "sessions/forecast-ensembles.html#visualisation-2",
+ "title": "Forecast ensembles",
+ "section": "Visualisation",
+ "text": "Visualisation\n\nSingle forecasts\nAgain we start by plotting a single forecast from each model and ensemble.\n\nweighted_ensembles |>\n filter(target_day == 57) |>\n plot_ensembles(onset_df |> filter(day >= 57, day <= 57 + 14))\n\n\n\n\n\n\n\n\nand on the log scale.\n\nweighted_ensembles |>\n filter(target_day == 57) |>\n plot_ensembles(onset_df |> filter(day >= 57, day <= 57 + 14)) +\n scale_y_log10()\n\n\n\n\n\n\n\n\n\n\nMultiple forecasts\nAs before we can plot a range of forecasts from each model and ensemble.\n\nplot_multiple_weighted_forecasts <- weighted_ensembles |>\n plot_ensembles(onset_df |> filter(day >= 21)) +\n lims(y = c(0, 400))\nplot_multiple_weighted_forecasts\n\n\n\n\n\n\n\n\nand on the log scale.\n\nplot_multiple_weighted_forecasts +\n scale_y_log10()\n\nScale for y is already present.\nAdding another scale for y, which will replace the existing scale.\n\n\nWarning in scale_y_log10(): log-10 transformation introduced infinite values.\nlog-10 transformation introduced infinite values.\nlog-10 transformation introduced infinite values.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nHow do the weighted ensembles compare to the simple ensembles? Which do you think is best? Are you surprised by the results? Can you think of any reasons that would explain them?\n\n\n\n\nModels weights\nNow that we have a weighted ensemble, we can also look at the weights of the individual models. Here we do this for the quantile regression averaging ensemble but we could also do this for the inverse WIS ensemble or any other weighted ensemble (for an unweighted ensemble the weights are all equal).\n\nqra_weights |>\n ggplot(aes(x = target_day, y = weight, fill = model)) +\n geom_col(position = \"stack\") +\n theme(legend.position = \"bottom\")\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nHow do the weights change over time? Are you surprised by the results given what you know about the models performance?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nHow do the weights change over time?\n\nEarly on the more statistical models have higher weights\nGradually the random walk model gains weight and by the end of the forecast horizon it represents the entire ensemble.\nNear the peak the mechanistic model also gains weight.\n\nAre you surprised by the results given what you know about the models performance?\n\nAs the random walk model is performing poorly, you would expect it to have low weights but actually it often doesn’t. This implies that its poor performance is restricted to certain parts of the outbreak.\nEven though the mechanistic model performs really well overall it is only included in the ensemble around the peak. This could be because the training data doesn’t include changes in the epidemic dynamics and so the mechanistic model is not given sufficient weight.",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-ensembles.html#evaluation-2",
+ "href": "sessions/forecast-ensembles.html#evaluation-2",
+ "title": "Forecast ensembles",
+ "section": "Evaluation",
+ "text": "Evaluation\nFor a final evaluation we can look at the scores for each model and ensemble again. We remove the two weeks of forecasts as these do not have a quantile regression average forecasts as these require training data to estimate.\n\nweighted_ensemble_scores <- weighted_ensembles |>\n filter(target_day >= 29) |>\n as_forecast_quantile(forecast_unit = c(\"target_day\", \"horizon\", \"model\")) |>\n score()\n\nAgain we can get a high level overview of the scores by model.\n\nweighted_ensemble_scores |>\n summarise_scores(by = c(\"model\"))\n\n model wis overprediction underprediction\n <char> <num> <num> <num>\n1: Inverse WIS ensemble 8.473266 4.422306 1.1072491\n2: Quantile Regression Average 7.362562 3.040759 1.9483875\n3: QRA by horizon 7.963665 3.409738 2.0659589\n4: Median filtered ensemble 7.022126 2.985429 1.2601905\n5: Mean ensemble 8.612259 4.452825 0.9873016\n6: Median ensemble 10.681557 6.238667 1.0547619\n7: Random walk 12.695476 7.961143 0.7709524\n8: More statistical 11.263910 6.030190 1.4395238\n9: More mechanistic 5.603486 1.571524 2.2731429\n dispersion bias interval_coverage_50 interval_coverage_90 ae_median\n <num> <num> <num> <num> <num>\n1: 2.943712 0.16619048 0.5095238 0.8190476 12.681390\n2: 2.373415 0.07190476 0.5047619 0.7952381 11.193606\n3: 2.487968 0.13367347 0.4897959 0.7755102 12.005322\n4: 2.776507 0.11476190 0.5523810 0.8761905 10.728571\n5: 3.172132 0.16571429 0.5333333 0.8714286 13.020635\n6: 3.388129 0.12238095 0.5476190 0.8714286 16.178571\n7: 3.963381 0.19571429 0.5142857 0.8523810 19.085714\n8: 3.794195 -0.04238095 0.5333333 0.8523810 16.780952\n9: 1.758819 0.17809524 0.4952381 0.7857143 8.685714\n\n\nRemembering the session on forecast evaluation, we should also check performance on the log scale.\n\nlog_ensemble_scores <- weighted_ensembles |>\n filter(target_day >= 29) |>\n as_forecast_quantile(forecast_unit = c(\"target_day\", \"horizon\", \"model\")) |>\n transform_forecasts(\n fun = log_shift,\n offset = 1,\n append = FALSE\n ) |>\n score()\n\nlog_ensemble_scores |>\n summarise_scores(by = c(\"model\"))\n\n model wis overprediction underprediction\n <char> <num> <num> <num>\n1: Inverse WIS ensemble 0.1738246 0.08278544 0.03090237\n2: Quantile Regression Average 0.1770130 0.08663305 0.03016151\n3: QRA by horizon 0.1657445 0.09253210 0.02377453\n4: Median filtered ensemble 0.1549876 0.07138677 0.02566016\n5: Mean ensemble 0.1689778 0.07928707 0.02702292\n6: Median ensemble 0.1889663 0.08160408 0.03772979\n7: Random walk 0.2046647 0.09644129 0.03538436\n8: More statistical 0.2160592 0.06803280 0.06550430\n9: More mechanistic 0.1598201 0.09859516 0.02048076\n dispersion bias interval_coverage_50 interval_coverage_90 ae_median\n <num> <num> <num> <num> <num>\n1: 0.06013681 0.16619048 0.5095238 0.8190476 0.2634627\n2: 0.06021847 0.07190476 0.5047619 0.7952381 0.2597085\n3: 0.04943783 0.13826531 0.4948980 0.7755102 0.2405630\n4: 0.05794062 0.11476190 0.5523810 0.8761905 0.2364991\n5: 0.06266781 0.16571429 0.5333333 0.8714286 0.2565306\n6: 0.06963244 0.12238095 0.5476190 0.8714286 0.2876486\n7: 0.07283902 0.19571429 0.5142857 0.8523810 0.3108524\n8: 0.08252213 -0.04238095 0.5333333 0.8523810 0.3193459\n9: 0.04074423 0.17809524 0.4952381 0.7857143 0.2396753\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nHow do the weighted ensembles compare to the simple ensembles on the natural and log scale?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nThe best ensembles slightly outperform some of the simple ensembles but there is no obvious benefit from using weighted ensembles. Why might this be the case?",
+ "crumbs": [
+ "Forecast ensembles"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation.html",
+ "href": "sessions/forecast-evaluation.html",
+ "title": "Forecast evaluation",
+ "section": "",
+ "text": "So far we have focused on visualising forecasts, including confronting them with events that were observed after the forecast was made. Besides visualising the forecasts, we can also summarise performance quantitatively. In this session you will get to know several ways of assessing different aspects of forecast performance.\n\n\n\nForecast evaluation\n\n\n\n\nThe aim of this session is to introduce the concept of forecast evaluation using scoring rules.\n\n\n\n\n\n\nSetup\n\n\n\n\n\n\n\nThe source file of this session is located at sessions/forecast-evaluation.qmd.\n\n\n\nIn this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and package for data wrangling, ggplot2 library for plotting and the scoringutils package for evaluating forecasts.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"ggplot2\")\nlibrary(\"scoringutils\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.\n\n\n\n\n\nWe set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Forecast evaluation"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation.html#slides",
+ "href": "sessions/forecast-evaluation.html#slides",
+ "title": "Forecast evaluation",
+ "section": "",
+ "text": "Forecast evaluation",
+ "crumbs": [
+ "Forecast evaluation"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation.html#objectives",
+ "href": "sessions/forecast-evaluation.html#objectives",
+ "title": "Forecast evaluation",
+ "section": "",
+ "text": "The aim of this session is to introduce the concept of forecast evaluation using scoring rules.\n\n\n\n\n\n\nSetup\n\n\n\n\n\n\n\nThe source file of this session is located at sessions/forecast-evaluation.qmd.\n\n\n\nIn this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and package for data wrangling, ggplot2 library for plotting and the scoringutils package for evaluating forecasts.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"ggplot2\")\nlibrary(\"scoringutils\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.\n\n\n\n\n\nWe set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Forecast evaluation"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation.html#source-file",
+ "href": "sessions/forecast-evaluation.html#source-file",
+ "title": "Forecast evaluation",
+ "section": "",
+ "text": "The source file of this session is located at sessions/forecast-evaluation.qmd.",
+ "crumbs": [
+ "Forecast evaluation"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation.html#libraries-used",
+ "href": "sessions/forecast-evaluation.html#libraries-used",
+ "title": "Forecast evaluation",
+ "section": "",
+ "text": "In this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and package for data wrangling, ggplot2 library for plotting and the scoringutils package for evaluating forecasts.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"ggplot2\")\nlibrary(\"scoringutils\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.",
+ "crumbs": [
+ "Forecast evaluation"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation.html#initialisation",
+ "href": "sessions/forecast-evaluation.html#initialisation",
+ "title": "Forecast evaluation",
+ "section": "",
+ "text": "We set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Forecast evaluation"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation.html#scoring-the-forecasts",
+ "href": "sessions/forecast-evaluation.html#scoring-the-forecasts",
+ "title": "Forecast evaluation",
+ "section": "Scoring the forecasts",
+ "text": "Scoring the forecasts\nWe now summarise performance quantitatively by using scoring metrics. Whilst some of these metrics are more useful for comparing models, many can be also be useful for understanding the performance of a single model.\n\n\n\n\n\n\nTip\n\n\n\nIn this session, we’ll use “proper” scoring rules: these are scoring rules that make sure no model can get better scores than the true model, i.e. the model used to generate the data. Of course we usually don’t know this (as we don’t know the “true model” for real-world data) but proper scoring rules incentivise forecasters to make their best attempt at reproducing its behaviour. For a comprehensive text on proper scoring rules and their mathematical properties, we recommend the classic paper by Gneiting and Raftery (2007).\n\n\nWe will use the {scoringutils} package to calculate these metrics. Our first step is to convert our forecasts into a format that the {scoringutils} package can use. We will use as_forecast_sample() to do this:\n\nsc_forecasts <- rw_forecasts |>\n left_join(onset_df, by = \"day\") |>\n filter(!is.na(.value)) |>\n as_forecast_sample(\n forecast_unit = c(\n \"target_day\", \"horizon\", \"model\"\n ),\n observed = \"onsets\",\n predicted = \".value\",\n sample_id = \".draw\"\n )\nsc_forecasts\n\nForecast type: sample\n\n\nForecast unit:\n\n\ntarget_day, horizon, and model\n\n\n\n sample_id predicted observed target_day horizon model\n <int> <num> <int> <num> <int> <char>\n 1: 1 4 4 22 1 Random walk\n 2: 2 2 4 22 1 Random walk\n 3: 3 2 4 22 1 Random walk\n 4: 4 6 4 22 1 Random walk\n 5: 5 2 4 22 1 Random walk\n --- \n223996: 996 1 2 127 14 Random walk\n223997: 997 0 2 127 14 Random walk\n223998: 998 0 2 127 14 Random walk\n223999: 999 1 2 127 14 Random walk\n224000: 1000 0 2 127 14 Random walk\n\n\nAs you can see this has created a forecast object which has a print method that summarises the forecasts.\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nWhat important information is in the forecast object?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\n\nThe forecast unit which is the target day, horizon, and model\nThe type of forecast which is a sample forecast\n\n\n\n\nEverything seems to be in order. We can now use the scoringutils package to calculate some metrics. We will use the default sample metrics (as our forecasts are in sample format) and score our forecasts.\n\nsc_scores <- sc_forecasts |>\n score()\n\nsc_scores\n\n target_day horizon model bias dss crps overprediction\n <num> <int> <char> <num> <num> <num> <num>\n 1: 22 1 Random walk -0.304 1.587880 0.623424 0.000\n 2: 22 2 Random walk 0.828 3.215491 1.664522 1.118\n 3: 22 3 Random walk 0.002 1.936078 0.574624 0.000\n 4: 22 4 Random walk 0.687 3.231981 1.570469 0.862\n 5: 22 5 Random walk 0.767 3.750147 2.172579 1.386\n --- \n220: 127 10 Random walk -0.821 3.120249 1.745842 0.000\n221: 127 11 Random walk -0.867 3.853982 1.972866 0.000\n222: 127 12 Random walk -0.949 7.803330 3.092738 0.000\n223: 127 13 Random walk -0.914 5.003912 2.318689 0.000\n224: 127 14 Random walk -0.637 1.140452 0.774974 0.000\n underprediction dispersion log_score mad ae_median se_mean\n <num> <num> <num> <num> <num> <num>\n 1: 0.154 0.469424 1.887790 1.4826 1 0.367236\n 2: 0.000 0.546522 2.231108 2.9652 3 8.479744\n 3: 0.000 0.574624 1.862234 2.9652 0 0.142884\n 4: 0.000 0.708469 2.175798 2.9652 3 9.284209\n 5: 0.000 0.786579 2.489797 2.9652 3 15.327225\n --- \n220: 1.400 0.345842 2.694333 1.4826 3 5.597956\n221: 1.684 0.288866 2.871442 1.4826 3 6.687396\n222: 2.832 0.260738 4.339301 1.4826 4 13.883076\n223: 2.062 0.256689 3.239324 1.4826 3 8.265625\n224: 0.496 0.278974 1.814469 1.4826 1 0.948676\n\n\n\n\n\n\n\n\nLearning more about the output of score()\n\n\n\n\n\nSee the documentation for get_metrics.forecast_sample()[https://epiforecasts.io/scoringutils/reference/get_metrics.forecast_sample.html] for information on the default metrics for forecasts that are represented as samples (in our case the samples generated by the stan model).\n\n\n\n\nAt a glance\nBefore we look in detail at the scores, we can use summarise_scores to get a quick overview of the scores. Don’t worry if you don’t understand all the scores yet, we will go some of them in more detail in the next section and you can find more information in the {scoringutils} documentation.\n\nsc_scores |>\n summarise_scores(by = \"model\")\n\n model bias dss crps overprediction underprediction\n <char> <num> <num> <num> <num> <num>\n1: Random walk 0.2147143 6.366294 13.50026 8.38367 0.7640446\n dispersion log_score mad ae_median se_mean\n <num> <num> <num> <num> <num>\n1: 4.352542 3.973451 18.2512 18.20982 1567.78\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nBefore we look in detail at the scores, what do you think the scores are telling you?\n\n\n\n\nContinuous ranked probability score\n\nWhat is the Continuous Ranked Probability Score (CRPS)?\nThe Continuous Ranked Probability Score (CRPS) is a proper scoring rule used to evaluate the accuracy of probabilistic forecasts. It is a generalization of the Mean Absolute Error (MAE) to probabilistic forecasts, where the forecast is a distribution rather than a single point estimate (i.e. like ours).\nThe CRPS can be thought about as the combination of two key aspects of forecasting: 1. The accuracy of the forecast in terms of how close the predicted values are to the observed value. 2. The confidence of the forecast in terms of the spread of the predicted values.\nBy balancing these two aspects, the CRPS provides a comprehensive measure of the quality of probabilistic forecasts.\n\n\n\n\n\n\nKey things to note about the CRPS\n\n\n\n\nSmall values are better\nAs it is an absolute scoring rule it can be difficult to use to compare forecasts across scales.\n\n\n\n\n\n\n\n\n\nMathematical Definition (optional)\n\n\n\n\n\nFor distributions with a finite first moment (a mean exists and it is finite), the CRPS can be expressed as:\n\\[\nCRPS(D, y) = \\mathbb{E}_{X \\sim D}[|X - y|] - \\frac{1}{2} \\mathbb{E}_{X, X' \\sim D}[|X - X'|]\n\\]\nwhere \\(X\\) and \\(X'\\) are independent random variables sampled from the distribution \\(D\\). To calculate this we simply replace \\(X\\) and \\(X'\\) by samples from our posterior distribution and sum over all possible combinations.\nThis equation can be broke down into the two components:\n\nBreakdown of the Components\n\nExpected Absolute Error Between Forecast and Observation: \\(\\mathbb{E}_{X \\sim D}[|X - y|]\\) This term represents the average absolute difference between the values predicted by the forecasted distribution \\(D\\) and the actual observed value \\(y\\). It measures how far, on average, the forecasted values are from the observed value. A smaller value indicates that the forecasted distribution is closer to the observed value.\nExpected Absolute Error Between Two Forecasted Values: \\(\\frac{1}{2} \\mathbb{E}_{X, X' \\sim D}[|X - X'|]\\) This term represents the average absolute difference between two independent samples from the forecasted distribution \\(D\\). It measures the internal variability or spread of the forecasted distribution. A larger value indicates a wider spread of the forecasted values.\n\n\n\nInterpretation\n\nFirst Term (\\(\\mathbb{E}_{X \\sim D}[|X - y|]\\)): This term penalizes the forecast based on how far the predicted values are from the observed value. It ensures that the forecast is accurate in terms of proximity to the actual observation.\nSecond Term (\\(\\frac{1}{2} \\mathbb{E}_{X, X' \\sim D}[|X - X'|]\\)): This term accounts for the spread of the forecasted distribution. It penalizes forecasts that are too uncertain or have a wide spread. By subtracting this term, the CRPS rewards forecasts that are not only accurate but also confident (i.e., have a narrow spread).\n\n\n\n\n\nWhilst the CRPS is a very useful metric it can be difficult to interpret in isolation. It is often useful to compare the CRPS of different models or to compare the CRPS of the same model under different conditions. For example, lets compare the CRPS across different forecast horizons.\n\nsc_scores |>\n summarise_scores(by = \"horizon\") |>\n ggplot(aes(x = horizon, y = crps)) +\n geom_point() +\n labs(title = \"CRPS by daily forecast horizon\", \n subtitle = \"Summarised across all forecasts\")\n\n\n\n\n\n\n\n\nand at different time points.\n\nsc_scores |>\n summarise_scores(by = \"target_day\") |>\n ggplot(aes(x = target_day, y = crps)) +\n geom_point() +\n labs(title = \"CRPS by forecast start date\", \n subtitle = \"Summarised across all forecasts\", x = \"forecast date\")\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nHow do the CRPS scores change based on forecast date? How do the CRPS scores change with forecast horizon? What does this tell you about the model?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\n\nThe CRPS scores increase for forecast dates where incidence is higher.\nThe CRPS scores increase with forecast horizon.\nAs the CRPS is an absolute measure it is hard to immediately know if the CRPS increasing with forecast date indicates that the model is performing worse.\nHowever, the CRPS increasing with forecast horizon is a sign that the model is struggling to capture the longer term dynamics of the epidemic.\n\n\n\n\n\n\n\nPIT histograms\nAs well as the CRPS we can also look at the calibration and bias of the model. Calibration is the agreement between the forecast probabilities and the observed frequencies. Bias is a measure of how likely the model is to over or under predict the observed values.\nThere are many ways to assess calibration and bias but one common way is to use a probability integral transform (PIT) histogram. This is a histogram of the cumulative distribution of function of a forecast evaluated at the observed value.\n\n\n\n\n\n\nInterpreting the PIT histogram\n\n\n\n\nIdeally PIT histograms should be uniform.\nIf is a U shape then the model is overconfident and if it is an inverted U shape then the model is underconfident.\nIf it is skewed then the model is biased towards the direction of the skew.\n\n\n\n\n\n\n\n\n\nMathematical Definition (optional)\n\n\n\n\n\nContinuous Case\nFor a continuous random variable \\(X\\) with cumulative distribution function (CDF) \\(F_X\\), the PIT is defined as:\n\\[\nY = F_X(X)\n\\]\nwhere \\(Y\\) is uniformly distributed on \\([0, 1]\\).\n\nInteger Case\nWhen dealing with integer forecasts, the standard PIT does not yield a uniform distribution even if the forecasts are perfectly calibrated. To address this, a randomized version of the PIT can be used (Czado, Gneiting, and Held 2009). For an integer-valued random variable \\(X\\) with CDF \\(F_X\\), the randomized PIT is defined as:\n\\[\nU = F_X(k) + v \\cdot (F_X(k) - F_X(k-1))\n\\]\nwhere:\n\n\\(k\\) is the observed integer value.\n\\(F_X(k)\\) is the CDF evaluated at \\(k\\).\n\\(v\\) is a random variable uniformly distributed on \\([0, 1]\\).\n\nThis transformation ensures that \\(U\\) is uniformly distributed on \\([0, 1]\\) if the forecasted distribution \\(F_X\\) is correctly specified.\n\n\n\n\nLet’s first look at the overall PIT histogram.\n\nsc_forecasts |>\n get_pit_histogram() |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n labs(title = \"PIT histogram\", x = \"Quantile\", y = \"Density\")\n\n\n\n\n\n\n\n\nAs before lets look at the PIT histogram by forecast horizon. To save space we will group horizons into a few days each:\n\nsc_forecasts |>\n mutate(group_horizon = case_when(\n horizon <= 3 ~ \"1-3\",\n horizon <= 7 ~ \"4-7\",\n horizon <= 14 ~ \"8-14\"\n )) |>\n get_pit_histogram(by = \"group_horizon\") |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() + \n facet_wrap(~group_horizon) +\n labs(title = \"PIT by forecast horizon (days)\")\n\n\n\n\n\n\n\n\nand then for different forecast dates.\n\nsc_forecasts |>\n get_pit_histogram(by = \"target_day\") |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() + \n facet_wrap(~target_day) +\n labs(title = \"PIT by forecast date\")\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nWhat do you think of the PIT histograms? Do they look well calibrated? Do they look biased?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\n\nIt looks like the model is biased towards overpredicting and that this bias gets worse at longer forecast horizons.\nLooking over forecast dates it looks like much of this bias is coming from near the outbreak peak where the model is consistently overpredicting but the model is also over predicting at other times.",
+ "crumbs": [
+ "Forecast evaluation"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation.html#scoring-on-the-log-scale",
+ "href": "sessions/forecast-evaluation.html#scoring-on-the-log-scale",
+ "title": "Forecast evaluation",
+ "section": "Scoring on the log scale",
+ "text": "Scoring on the log scale\nWe can also score on the logarithmic scale. This can be useful if we are interested in the relative performance of the model at different scales of the data, for example if we are interested in the model’s performance at capturing the exponential growth phase of the epidemic. In some sense scoring in this way can be an approximation of scoring the effective reproduction number estimates. Doing this directly can be difficult as the effective reproduction number is a latent variable and so we cannot directly score it.\nWe again use scoringutils but first transform both the forecasts and observations to the log scale.\n\nlog_sc_forecasts <- sc_forecasts |>\n transform_forecasts(\n fun = log_shift,\n offset = 1,\n append = FALSE\n )\n\nlog_scores <- log_sc_forecasts |>\n score()\n\nFor more on scoring on the log scale see the paper by Bosse et al. (2023).\n\nAt a glance\n\nlog_scores |>\n summarise_scores(by = \"model\")\n\n model bias dss crps overprediction underprediction\n <char> <num> <num> <num> <num> <num>\n1: Random walk 0.1774554 -0.6084371 0.2451112 0.121214 0.03678654\n dispersion log_score mad ae_median se_mean\n <num> <num> <num> <num> <num>\n1: 0.08711065 0.5440927 0.3761217 0.3379461 0.2082002\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nBefore we look in detail at the scores, what do you think the scores are telling you? How do you think they will differ from the scores on the natural scale?\n\n\n\n\nCRPS\n\nlog_scores |>\n summarise_scores(by = \"horizon\") |>\n ggplot(aes(x = horizon, y = crps)) +\n geom_point() +\n labs(title = \"CRPS by daily forecast horizon, scored on the log scale\")\n\n\n\n\n\n\n\n\nand across different forecast dates\n\nlog_scores |>\n summarise_scores(by = \"target_day\") |>\n ggplot(aes(x = target_day, y = crps)) +\n geom_point() +\n labs(title = \"CRPS by forecast date, scored on the log scale\")\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nHow do the CRPS scores change based on forecast date? How do the CRPS scores change with forecast horizon? What does this tell you about the model?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\n\nAs for the natural scale CRPS scores increase with forecast horizon but now the increase appears to be linear vs exponential.\nThere has been a reduction in the CRPS scores for forecast dates near the outbreak peak compared to other forecast dates but this is still the period where the model is performing worst.\n\n\n\n\n\n\nPIT histograms\nLet’s first look at the overall PIT histogram.\n\nlog_sc_forecasts |>\n get_pit_histogram(by = \"model\") |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n labs(title = \"PIT histogram, scored on the log scale\")\n\n\n\n\n\n\n\n\nAs before lets look at the PIT histogram by forecast horizon\n\nlog_sc_forecasts |>\n mutate(group_horizon = case_when(\n horizon <= 3 ~ \"1-3\",\n horizon <= 7 ~ \"4-7\",\n horizon <= 14 ~ \"8-14\"\n )) |>\n get_pit_histogram(by = \"group_horizon\") |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n facet_wrap(~group_horizon) +\n labs(title = \"PIT by forecast horizon, scored on the log scale\")\n\n\n\n\n\n\n\n\nand then for different forecast dates.\n\nlog_sc_forecasts |>\n get_pit_histogram(by = \"target_day\") |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n facet_wrap(~target_day) +\n labs(title = \"PIT by forecast date, scored on the log scale\")\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nWhat do you think of the PIT histograms? Do they look well calibrated? Do they look biased?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\n\nThe overall PIT histograms suggest that the model is less biased to over predict when scored on the log scale than the natural scale, but it is still biased. This makes sense when we think back to the comparison of reproduction number estimates and forecasts we made earlier where the model was consistently over predicting on the reproduction number.\nBy forecast horizon the model is still biased towards over predicting but this bias is less pronounced than on the natural scale.\nTowards the end and beginning of the forecast period the model appears to be well calibrated on the log scale but is biased towards over predicting in the middle of the forecast period.\nThis matches with our knowledge of the underlying reproduction number which were initially constant and then began to decrease only to stabilise towards the end of the outbreak.",
+ "crumbs": [
+ "Forecast evaluation"
+ ]
+ },
+ {
+ "objectID": "index.html",
+ "href": "index.html",
+ "title": "Understanding, evaluating, and improving forecasts of infectious disease burden",
+ "section": "",
+ "text": "This course is a shorter version of a course on Nowcasting and forecasting infectious disease dynamics.\nAll materials here are provided under the permissive MIT License."
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#ensembles",
+ "href": "sessions/slides/introduction-to-ensembles.html#ensembles",
+ "title": "Introduction to ensembles",
+ "section": "Ensembles",
+ "text": "Ensembles\n\nCombine many different models’ forecasts into a single prediction"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#why-ensemble",
+ "href": "sessions/slides/introduction-to-ensembles.html#why-ensemble",
+ "title": "Introduction to ensembles",
+ "section": "Why ensemble?",
+ "text": "Why ensemble?\n\nMany uncertainties, many approaches: many models\n\nLayers of uncertainty\n\nModel parameters\n\ne.g. parameterising delay distributions\n\nModel structure\n\ne.g. more mechanistic or more statistical approaches\n\n\nWant to use all available information"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#why-ensemble-1",
+ "href": "sessions/slides/introduction-to-ensembles.html#why-ensemble-1",
+ "title": "Introduction to ensembles",
+ "section": "Why ensemble?",
+ "text": "Why ensemble?\n“Whole is greater than sum of parts”\n\nAverage of multiple predictions is often more performant than any individual model\n\nHistory in weather & economic forecasting\nSeen this in infectious disease forecasting\n\n“Forecast challenges”: Ebola, dengue, flu, COVID-19…"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#ensemble-methods",
+ "href": "sessions/slides/introduction-to-ensembles.html#ensemble-methods",
+ "title": "Introduction to ensembles",
+ "section": "Ensemble methods",
+ "text": "Ensemble methods\n\n\nSummarising across models to create single (probabilistic) prediction\n\ne.g. average at each models’ probabilistic quantiles\n\nMean\nMedian - trims the outliers, so narrows the uncertainty"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#ensemble-methods-1",
+ "href": "sessions/slides/introduction-to-ensembles.html#ensemble-methods-1",
+ "title": "Introduction to ensembles",
+ "section": "Ensemble methods",
+ "text": "Ensemble methods\n\nEqual or weighted combination\n\nWeight models by past forecast performance\n\ne.g. using forecast scores\n\nRarely better than equal average"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#collaborative-modelling",
+ "href": "sessions/slides/introduction-to-ensembles.html#collaborative-modelling",
+ "title": "Introduction to ensembles",
+ "section": "Collaborative modelling",
+ "text": "Collaborative modelling"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#section",
+ "href": "sessions/slides/introduction-to-ensembles.html#section",
+ "title": "Introduction to ensembles",
+ "section": "",
+ "text": "… European Respicast"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#single-model",
+ "href": "sessions/slides/introduction-to-ensembles.html#single-model",
+ "title": "Introduction to ensembles",
+ "section": "Single model",
+ "text": "Single model"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#multiple-models",
+ "href": "sessions/slides/introduction-to-ensembles.html#multiple-models",
+ "title": "Introduction to ensembles",
+ "section": "… Multiple models",
+ "text": "… Multiple models"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#multi-model-ensemble",
+ "href": "sessions/slides/introduction-to-ensembles.html#multi-model-ensemble",
+ "title": "Introduction to ensembles",
+ "section": "… … Multi-model ensemble",
+ "text": "… … Multi-model ensemble"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-ensembles.html#your-turn",
+ "href": "sessions/slides/introduction-to-ensembles.html#your-turn",
+ "title": "Introduction to ensembles",
+ "section": " Your Turn",
+ "text": "Your Turn\n\nCreate unweighted and weighted ensembles using forecasts from multiple models.\nEvaluate the forecasts from ensembles compared to their constituent models."
+ },
+ {
+ "objectID": "sessions/slides/forecast-evaluation.html#your-turn",
+ "href": "sessions/slides/forecast-evaluation.html#your-turn",
+ "title": "Forecast evaluation",
+ "section": " Your Turn",
+ "text": "Your Turn\n\nLoad forecasts from the model we have visualised previously.\nEvaluate the forecasts using proper scoring rules"
+ },
+ {
+ "objectID": "sessions/introduction-and-course-overview.html",
+ "href": "sessions/introduction-and-course-overview.html",
+ "title": "Introduction and course overview",
+ "section": "",
+ "text": "Introduction to the course and the instructors\n\nAim of the course\nIn this course we will explore how to:\n\nVisualize and interpret infectious disease forecasts\nEvaluate forecast performance and limitations\nCombine multiple forecasts into ensembles\n\nPredictive modeling has become crucial for public health decision-making and pandemic preparedness. Through this workshop, we aim to make forecast interpretation and evaluation more accessible to academics and public health institutions.\n\n\nApproach\nEach session in the course:\n\nbuilds on the previous one so that participants will have an overview of forecast evaluation by the end of the course;\nstarts with a short introductory talk;\nmainly consists of interactive content that participants will work through;\nhas optional/additional material that can be skipped or completed after the course ends;\n\nFor those attending the in-person version the course also:\n\nhas multiple instructors ready to answer questions about this content; if several people have a similar question we may pause the session and discuss it with the group;\nends with a wrap-up and discussion where we review the sessions material.\n\n\n\nTimeline for the course\nThe course consists of four main sessions spread across an afternoon. If you are attending the in-person version of the course, the schedule is available here. If you are studying this material on your own using the website, you can go through it at your own pace.\nLet’s get started!\n\nHave a more detailed look at the learning objectives\nIf you haven’t already, start with getting set up for the course\nStart the course by learning about Visualising infectious disease forecasts",
+ "crumbs": [
+ "Introduction and course overview"
+ ]
+ },
+ {
+ "objectID": "sessions.html",
+ "href": "sessions.html",
+ "title": "Sessions",
+ "section": "",
+ "text": "13:30-13:35\n\nIntroduction to the course and the instructors (5 mins)"
+ },
+ {
+ "objectID": "sessions.html#session-0-introduction-and-course-overview",
+ "href": "sessions.html#session-0-introduction-and-course-overview",
+ "title": "Sessions",
+ "section": "",
+ "text": "13:30-13:35\n\nIntroduction to the course and the instructors (5 mins)"
+ },
+ {
+ "objectID": "sessions.html#session-1-forecasting",
+ "href": "sessions.html#session-1-forecasting",
+ "title": "Sessions",
+ "section": "Session 1: Forecasting",
+ "text": "Session 1: Forecasting\n13:35-14:10\n\nIntroduction to forecasting as an epidemiological problem (10 mins)\nPractice session: Visualising infectious disease forecasts (25 minutes)"
+ },
+ {
+ "objectID": "sessions.html#session-2-forecast-evaluation",
+ "href": "sessions.html#session-2-forecast-evaluation",
+ "title": "Sessions",
+ "section": "Session 2: Forecast evaluation",
+ "text": "Session 2: Forecast evaluation\n14:10-15:00\n\nAn introduction to forecast evaluation (5 mins)\nPractice session: Evaluating forecasts from a range of models (40 mins)\nWrap up (5 mins)"
+ },
+ {
+ "objectID": "sessions.html#session-3-evaluating-forecasts-from-multiple-models",
+ "href": "sessions.html#session-3-evaluating-forecasts-from-multiple-models",
+ "title": "Sessions",
+ "section": "Session 3: Evaluating forecasts from multiple models",
+ "text": "Session 3: Evaluating forecasts from multiple models\n15:10-15:50\n\nWhy might we want to evaluate forecasts from multiple models? (5 mins)\nPractice session: Evaluating forecasts from multiple models (40 mins)\nWrap up (5 mins)\n\nThere is a coffee break at 15:30-15:40 you are welcome to attend this or keep working through the course material."
+ },
+ {
+ "objectID": "sessions.html#session-4-forecast-ensembles",
+ "href": "sessions.html#session-4-forecast-ensembles",
+ "title": "Sessions",
+ "section": "Session 4: Forecast ensembles",
+ "text": "Session 4: Forecast ensembles\n15:50-17:10\n\nIntroduction to forecast ensembles (10 mins)\nPractice session: Creating forecast ensembles (60 mins)\nWrap up (10 mins)"
+ },
+ {
+ "objectID": "sessions.html#session-5-end-of-course-summary",
+ "href": "sessions.html#session-5-end-of-course-summary",
+ "title": "Sessions",
+ "section": "Session 5: End of course summary",
+ "text": "Session 5: End of course summary\n17:10-17:30\n\nSummary of the course (10 mins)\nFinal discussion and closing (10 mins)"
+ },
+ {
+ "objectID": "help.html",
+ "href": "help.html",
+ "title": "Getting help",
+ "section": "",
+ "text": "For any questions about the course or its content, feel free to use the Discussion board"
+ },
+ {
+ "objectID": "learning_objectives.html",
+ "href": "learning_objectives.html",
+ "title": "Independent learning outcomes",
+ "section": "",
+ "text": "understanding of forecasting as an epidemiological problem\nunderstanding of ways to visualise forecasts"
+ },
+ {
+ "objectID": "learning_objectives.html#forecasting",
+ "href": "learning_objectives.html#forecasting",
+ "title": "Independent learning outcomes",
+ "section": "",
+ "text": "understanding of forecasting as an epidemiological problem\nunderstanding of ways to visualise forecasts"
+ },
+ {
+ "objectID": "learning_objectives.html#evaluating-forecasts",
+ "href": "learning_objectives.html#evaluating-forecasts",
+ "title": "Independent learning outcomes",
+ "section": "Evaluating forecasts",
+ "text": "Evaluating forecasts\n\nfamiliarity with metrics for evaluating probabilistic forecasts and their properties\nability to score probabilistic forecasts in R"
+ },
+ {
+ "objectID": "learning_objectives.html#ensemble-models",
+ "href": "learning_objectives.html#ensemble-models",
+ "title": "Independent learning outcomes",
+ "section": "Ensemble models",
+ "text": "Ensemble models\n\nunderstanding of predictive ensembles and their properties\nability to create a predictive ensemble of forecasts in R\nunderstanding the concept of weighted forecast ensembles"
+ },
+ {
+ "objectID": "sessions/end-of-course-summary-and-discussion.html",
+ "href": "sessions/end-of-course-summary-and-discussion.html",
+ "title": "End of course summary and discussion",
+ "section": "",
+ "text": "We hope that you’ve enjoyed taking this course on understanding and evaluating forecasts of infectious disease burden. We will be very happy to have your feedback, comments or any questions on the course discussion page.\n\n\nEnd of course summary",
+ "crumbs": [
+ "End of course summary and discussion"
+ ]
+ },
+ {
+ "objectID": "sessions/end-of-course-summary-and-discussion.html#slides",
+ "href": "sessions/end-of-course-summary-and-discussion.html#slides",
+ "title": "End of course summary and discussion",
+ "section": "",
+ "text": "End of course summary",
+ "crumbs": [
+ "End of course summary and discussion"
+ ]
+ },
+ {
+ "objectID": "sessions/slides/closing.html#session-1-forecasting",
+ "href": "sessions/slides/closing.html#session-1-forecasting",
+ "title": "End of course summary",
+ "section": "Session 1: Forecasting",
+ "text": "Session 1: Forecasting\n\nforecasting is the task of making unconditional statements about the future\nmeaningful forecasts are probabilistic\nwe can use visualisation understand the predictive performance of a model"
+ },
+ {
+ "objectID": "sessions/slides/closing.html#session-2-forecast-evaluation",
+ "href": "sessions/slides/closing.html#session-2-forecast-evaluation",
+ "title": "End of course summary",
+ "section": "Session 2: Forecast evaluation",
+ "text": "Session 2: Forecast evaluation\n\nwe can assess forecasts using proper scoring rules\nwe can use scoring to quantify the predictive performance of different models"
+ },
+ {
+ "objectID": "sessions/slides/closing.html#session-3-forecast-ensembles",
+ "href": "sessions/slides/closing.html#session-3-forecast-ensembles",
+ "title": "End of course summary",
+ "section": "Session 3: Forecast ensembles",
+ "text": "Session 3: Forecast ensembles\n\nensembles can combine forecasts from multiple models\nsimple ensembles often outperform indivdiual models\nweighted ensembles can learn from past performance aiming to make better forecasts"
+ },
+ {
+ "objectID": "sessions/slides/forecasting-as-an-epidemiological-problem.html#your-turn",
+ "href": "sessions/slides/forecasting-as-an-epidemiological-problem.html#your-turn",
+ "title": "Forecasting as an epidemiological problem",
+ "section": " Your Turn",
+ "text": "Your Turn\n\nLoad some forecasts we have generated and visualise them alongside the data.\nExplore different ways of visualising forecasts to assess how good the model performs at forecasting."
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-the-spectrum-of-forecasting-models.html#different-types-of-models",
+ "href": "sessions/slides/introduction-to-the-spectrum-of-forecasting-models.html#different-types-of-models",
+ "title": "Introduction to the spectrum of forecasting models",
+ "section": "Different types of models",
+ "text": "Different types of models\n\n\n\n\nWe can classify models by the level of mechanism they include\nAll of the model types we will introduce in the next few slides have been used for COVID-19 forecasting (the US and/or European COVID-19 forecast hub)\n\nNOTE: level of mechanism \\(\\neq\\) model complexity\n\n\nCramer et al., Scientific Data, 2022"
+ },
+ {
+ "objectID": "sessions/slides/introduction-to-the-spectrum-of-forecasting-models.html#your-turn",
+ "href": "sessions/slides/introduction-to-the-spectrum-of-forecasting-models.html#your-turn",
+ "title": "Introduction to the spectrum of forecasting models",
+ "section": " Your Turn",
+ "text": "Your Turn\n\nLoad forecasts from the model we have visualised previously.\nEvaluate the forecasts using proper scoring rules"
+ },
+ {
+ "objectID": "getting-set-up.html",
+ "href": "getting-set-up.html",
+ "title": "Overview",
+ "section": "",
+ "text": "Each session in this course uses R code for demonstration. All the content is self-contained within a R software package designed for the course and uses notebooks for each session.\nIf you would like to you can follow the course material without installing any software or running any code. However, if you would like to run the code yourself, you will need to follow the instructions below."
+ },
+ {
+ "objectID": "getting-set-up.html#installation-of-the-nfidd-package",
+ "href": "getting-set-up.html#installation-of-the-nfidd-package",
+ "title": "Overview",
+ "section": "Installation of the nfidd package",
+ "text": "Installation of the nfidd package\nTo install the packages needed in the course, including the nfidd package that contains data files and helper functions used throughout, you can use the pak package:\n\noptions(repos = c(\n \"CRAN\" = \"https://cloud.r-project.org\",\n \"stan-dev\" = \"https://stan-dev.r-universe.dev\",\n \"epiforecasts\" = \"https://epiforecasts.r-universe.dev\"\n))\ninstall.packages(\"nfidd\", dependencies = TRUE)\n\nThen you can check that the installation completed successfully by loading the package into your R session:\n\nlibrary(\"nfidd\")"
+ },
+ {
+ "objectID": "sessions/forecast-visualisation.html",
+ "href": "sessions/forecast-visualisation.html",
+ "title": "Visualising infectious disease forecasts",
+ "section": "",
+ "text": "Epidemiological forecasts are probabilistic statements about what might happen to population disease burden in the future. In this session we will make some simple forecasts using a commonly used infectious disease model, based on the renewal equation. We will see how we can visualise such forecasts, and visually compare them to what really happened.\n\n\n\nForecasting as an epidemiological problem\n\n\n\n\nThe aim of this session is to introduce the concept of forecasting and forecast visualisation using a simple model.\n\n\n\n\n\n\nSetup\n\n\n\n\n\n\n\nThe source file of this session is located at sessions/forecasting-visualisation.qmd.\n\n\n\nIn this session we will use the nfidd package to load a data set of infection times and corresponding forecasts, the dplyr package for data wrangling, and the ggplot2 library for plotting.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"ggplot2\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.\n\n\n\n\n\nWe set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)\nWe will now look at some data from an infectious disease outbreak and forecasts from a renewal equation model. First, we simulate some data using functions from the nfidd package.\ngen_time_pmf <- make_gen_time_pmf()\nip_pmf <- make_ip_pmf()\nonset_df <- simulate_onsets(\n make_daily_infections(infection_times), gen_time_pmf, ip_pmf\n)\ntail(onset_df)\n\n# A tibble: 6 × 3\n day onsets infections\n <dbl> <int> <int>\n1 137 1 5\n2 138 3 1\n3 139 5 4\n4 140 3 1\n5 141 2 1\n6 142 2 0\nThis uses a data set of infection times that is included in the R package, and a function to turn these into daily number of observed symptom onsets. Do not worry too much about the details of this. The important thing is that we now have a data set onset_df of symptom onset.",
+ "crumbs": [
+ "Visualising infectious disease forecasts"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-visualisation.html#slides",
+ "href": "sessions/forecast-visualisation.html#slides",
+ "title": "Visualising infectious disease forecasts",
+ "section": "",
+ "text": "Forecasting as an epidemiological problem",
+ "crumbs": [
+ "Visualising infectious disease forecasts"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-visualisation.html#objectives",
+ "href": "sessions/forecast-visualisation.html#objectives",
+ "title": "Visualising infectious disease forecasts",
+ "section": "",
+ "text": "The aim of this session is to introduce the concept of forecasting and forecast visualisation using a simple model.\n\n\n\n\n\n\nSetup\n\n\n\n\n\n\n\nThe source file of this session is located at sessions/forecasting-visualisation.qmd.\n\n\n\nIn this session we will use the nfidd package to load a data set of infection times and corresponding forecasts, the dplyr package for data wrangling, and the ggplot2 library for plotting.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"ggplot2\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.\n\n\n\n\n\nWe set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Visualising infectious disease forecasts"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-visualisation.html#source-file",
+ "href": "sessions/forecast-visualisation.html#source-file",
+ "title": "Visualising infectious disease forecasts",
+ "section": "",
+ "text": "The source file of this session is located at sessions/forecasting-visualisation.qmd.",
+ "crumbs": [
+ "Visualising infectious disease forecasts"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-visualisation.html#libraries-used",
+ "href": "sessions/forecast-visualisation.html#libraries-used",
+ "title": "Visualising infectious disease forecasts",
+ "section": "",
+ "text": "In this session we will use the nfidd package to load a data set of infection times and corresponding forecasts, the dplyr package for data wrangling, and the ggplot2 library for plotting.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"ggplot2\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.",
+ "crumbs": [
+ "Visualising infectious disease forecasts"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-visualisation.html#initialisation",
+ "href": "sessions/forecast-visualisation.html#initialisation",
+ "title": "Visualising infectious disease forecasts",
+ "section": "",
+ "text": "We set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Visualising infectious disease forecasts"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-visualisation.html#simulating-a-geometric-random-walk",
+ "href": "sessions/forecast-visualisation.html#simulating-a-geometric-random-walk",
+ "title": "Visualising infectious disease forecasts",
+ "section": "Simulating a geometric random walk",
+ "text": "Simulating a geometric random walk\nYou can have a look at an R function for performing the geometric random walk:\n\ngeometric_random_walk\n\nfunction (init, noise, std) \n{\n n <- length(noise) + 1\n x <- numeric(n)\n x[1] <- init\n for (i in 2:n) {\n x[i] <- x[i - 1] + noise[i - 1] * std\n }\n return(exp(x))\n}\n<bytecode: 0x55a374bf3f40>\n<environment: namespace:nfidd>\n\n\nWe can use this function to simulate a random walk:\n\nR <- geometric_random_walk(init = 1, noise = rnorm(100), std = 0.1)\ndata <- tibble(t = seq_along(R), R = exp(R))\n\nggplot(data, aes(x = t, y = R)) +\n geom_line() +\n labs(title = \"Simulated data from a random walk model\",\n x = \"Time\",\n y = \"R\")\n\n\n\n\n\n\n\n\nYou can repeat this multiple times either with the same parameters or changing some to get a feeling for what this does.",
+ "crumbs": [
+ "Visualising infectious disease forecasts"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-visualisation.html#visualising-the-forecast",
+ "href": "sessions/forecast-visualisation.html#visualising-the-forecast",
+ "title": "Visualising infectious disease forecasts",
+ "section": "Visualising the forecast",
+ "text": "Visualising the forecast\nWe can now visualise a forecast made from the renewal equation model we used for forecasting. Once again, this forecast is provided in the nfidd package which we loaded earlier. We will first extract the forecast and then plot the forecasted number of symptom onsets alongside the observed number of symptom onsets before the forecast was made.\n\ndata(rw_forecasts)\nhorizon <- 14\nforecast_day <- 71\nforecast <- rw_forecasts |>\n ungroup() |>\n filter(target_day == forecast_day)\n\nprevious_onsets <- onset_df |>\n filter(day <= forecast_day)\n\n\n\n\n\n\n\nHow did we generate these forecasts?\n\n\n\n\n\nSome important things to note about these forecasts:\n\nWe used a 14 day forecast horizon.\nEach forecast used all the data up to the forecast date.\nWe generated 1000 predictive posterior samples for each forecast.\nWe started forecasting 3 weeks into the outbreak and then forecast every 7 days until the end of the data (excluding the last 14 days to allow a full forecast).\nWe use the same simulated outbreak data as before:\n\n\ngen_time_pmf <- make_gen_time_pmf()\nip_pmf <- make_ip_pmf()\nonset_df <- simulate_onsets(\n make_daily_infections(infection_times), gen_time_pmf, ip_pmf\n)\nhead(onset_df)\n\n# A tibble: 6 × 3\n day onsets infections\n <dbl> <int> <int>\n1 1 0 0\n2 2 0 1\n3 3 0 0\n4 4 0 2\n5 5 0 1\n6 6 0 1\n\n\n\n\n\n\nforecast |>\n filter(.draw %in% sample(.draw, 100)) |>\n ggplot(aes(x = day)) +\n geom_line(alpha = 0.1, aes(y = .value, group = .draw)) +\n geom_point(data = previous_onsets, aes(x = day, y = onsets), color = \"black\") +\n labs(\n title = \"Symptom onsets\",\n subtitle = \"Forecast (trajectories) and observed (points)\"\n )\n\n\n\n\n\n\n\n\nIn this plot, the dots show the data and the lines are forecast trajectories that the model deems plausible and consistent with the data so far.\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nWhat do you think of this forecast? Does it match your intuition of how this outbreak will continue? Is there another way you could visualise the forecast that might be more informative?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\n\nThe forecast mainly predicts things to continue growing as they have been. However, some of the trajectories are going down, indicating that with some probabilities the outbreak might end.\nBased purely on the data and without knowing anything else about the disease and setting, it is hard to come up with an alternative. The model seems sensible given the available data. In particular, uncertainty increases with increasing distance to the data, which is a sign of a good forecasting model.\nInstead of visualising plausible trajectories we could visualise a cone with a central forecast and uncertainty around it. We will look at this in the next session as an alternative.\n\n\n\n\nIf we want to know how well a model is doing at forecasting, we can later compare it do the data of what really happened. If we do this, we get the following plot.\n\nfuture_onsets <- onset_df |>\n filter(day <= forecast_day + horizon)\n\nforecast |>\n filter(.draw %in% sample(.draw, 100)) |>\n ggplot(aes(x = day)) +\n geom_line(alpha = 0.1, aes(y = .value, group = .draw)) +\n geom_point(data = future_onsets, aes(x = day, y = onsets), color = \"black\") +\n labs(\n title = \"Symptom onsets\",\n subtitle = \"Forecast (trajectories) and observed (points)\"\n )\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nWhat do you think now of this forecast? Did the model do a good job? Again, is there another way you could visualise the forecast that might be more informative?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\n\nOn the face of it the forecast looks very poor with some very high predictions compared to the data.\nBased on this visualisation it is hard to tell if the model is doing a good job but it seems like it is not.\nAs outbreaks are generally considered to be exponential processes it might be more informative to plot the forecast on the log scale.\n\n\nforecast |>\n filter(.draw %in% sample(.draw, 100)) |>\n ggplot(aes(x = day)) +\n geom_line(alpha = 0.1, aes(y = .value, group = .draw)) +\n geom_point(data = future_onsets, aes(x = day, y = onsets), color = \"black\") +\n scale_y_log10() +\n labs(title = \"Symptom onsets, log scale\", subtitle = \"Forecast and observed\")\n\nWarning in scale_y_log10(): log-10 transformation introduced infinite values.\n\n\n\n\n\n\n\n\n\nThis should be a lot more informative. We see that for longer forecast horizons the model is not doing a great job of capturing the reduction in symptom onsets. However, we can now see that the model seems to be producing very reasonable forecasts for the first week or so of the forecast. This is a common pattern in forecasting where a model is good at capturing the short term dynamics but struggles with the longer term dynamics.\n\n\n\nWe managed to learn quite a lot about our model’s forecasting limitations just looking at a single forecast using visualisations. However, what if we wanted to quantify how well the model is doing? In order to do that we need to look at multiple forecasts from the model.",
+ "crumbs": [
+ "Visualising infectious disease forecasts"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-visualisation.html#visualising-multiple-forecasts-from-a-single-model",
+ "href": "sessions/forecast-visualisation.html#visualising-multiple-forecasts-from-a-single-model",
+ "title": "Visualising infectious disease forecasts",
+ "section": "Visualising multiple forecasts from a single model",
+ "text": "Visualising multiple forecasts from a single model\nAs for a single forecast, our first step is to visualise the forecasts as this can give us a good idea of how well the model is doing without having to calculate any metrics.\n\nrw_forecasts |>\n filter(.draw %in% sample(.draw, 100)) |>\n ggplot(aes(x = day)) +\n geom_line(\n aes(y = .value, group = interaction(.draw, target_day), col = target_day),\n alpha = 0.1\n ) +\n geom_point(\n data = onset_df |>\n filter(day >= 21),\n aes(x = day, y = onsets), color = \"black\") +\n scale_color_binned(type = \"viridis\") +\n labs(title = \"Weekly forecasts of symptom onsets over an outbreak\",\n col = \"Forecast start day\")\n\n\n\n\n\n\n\n\nAs for the single forecast it may be helpful to also plot the forecast on the log scale.\n\nrw_forecasts |>\n filter(.draw %in% sample(.draw, 100)) |> \n ggplot(aes(x = day)) +\n geom_line(\n aes(y = .value, group = interaction(.draw, target_day), col = target_day),\n alpha = 0.1\n ) +\n geom_point(data = onset_df, aes(x = day, y = onsets), color = \"black\") +\n scale_y_log10() +\n scale_color_binned(type = \"viridis\") +\n labs(title = \"Weekly symptom onset forecasts: log scale\",\n col = \"Forecast start day\")\n\nWarning in scale_y_log10(): log-10 transformation introduced infinite values.\nlog-10 transformation introduced infinite values.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nWhat do you think of these forecasts? Are they any good? How well do they capture changes in trend? Does the uncertainty seem reasonable? Do they seem to under or over predict consistently? Would you visualise the forecast in a different way?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nWhat do you think of these forecasts?\n\nWe think these forecasts are a reasonable place to start but there is definitely room for improvement.\n\nAre they any good?\n\nThey seem to do a reasonable job of capturing the short term dynamics but struggle with the longer term dynamics.\n\nHow well do they capture changes in trend?\n\nThere is little evidence of the model capturing the reduction in onsets before it begins to show in the data.\n\nDoes the uncertainty seem reasonable?\n\nOn the natural scale it looks like the model often over predicts. Things seem more balanced on the log scale but the model still seems to be overly uncertain.\n\nDo they seem to under or over predict consistently?\n\nIt looks like the model is consistently over predicting on the natural scale but this is less clear on the log scale.",
+ "crumbs": [
+ "Visualising infectious disease forecasts"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation-of-multiple-models.html",
+ "href": "sessions/forecast-evaluation-of-multiple-models.html",
+ "title": "Evaluating forecasts from multiple models",
+ "section": "",
+ "text": "We can classify models along a spectrum by how much they include an understanding of underlying processes, or mechanisms; or whether they emphasise drawing from the data using a statistical approach. These different approaches all have different strength and weaknesses, and it is not clear a prior which one produces the best forecast in any given situation.\nIn this session, we’ll start with forecasts from models of different levels of mechanism vs. statistical complexity and evaluate them using visualisation and proper scoring rules as we did in the last session for the random walk model\n\n\n\nIntroduction to the spectrum of forecasting models\n\n\n\n\nThe aim of this session is to introduce the concept of a spectrum of forecasting models and to demonstrate how to evaluate a range of different models from across this spectrum.\n\n\n\n\n\n\nSetup\n\n\n\n\n\n\n\nThe source file of this session is located at sessions/forecast-ensembles.qmd.\n\n\n\nIn this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and tidyr packages for data wrangling, ggplot2 library for plotting, the tidybayes package for extracting results of the inference and the scoringutils package for evaluating forecasts. We will also use qra for quantile regression averaging in the weighted ensemble section.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"tidyr\")\nlibrary(\"ggplot2\")\nlibrary(\"scoringutils\")\nlibrary(\"qra\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.\n\n\n\n\n\nWe set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Evaluating forecasts from multiple models"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation-of-multiple-models.html#slides",
+ "href": "sessions/forecast-evaluation-of-multiple-models.html#slides",
+ "title": "Evaluating forecasts from multiple models",
+ "section": "",
+ "text": "Introduction to the spectrum of forecasting models",
+ "crumbs": [
+ "Evaluating forecasts from multiple models"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation-of-multiple-models.html#objectives",
+ "href": "sessions/forecast-evaluation-of-multiple-models.html#objectives",
+ "title": "Evaluating forecasts from multiple models",
+ "section": "",
+ "text": "The aim of this session is to introduce the concept of a spectrum of forecasting models and to demonstrate how to evaluate a range of different models from across this spectrum.\n\n\n\n\n\n\nSetup\n\n\n\n\n\n\n\nThe source file of this session is located at sessions/forecast-ensembles.qmd.\n\n\n\nIn this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and tidyr packages for data wrangling, ggplot2 library for plotting, the tidybayes package for extracting results of the inference and the scoringutils package for evaluating forecasts. We will also use qra for quantile regression averaging in the weighted ensemble section.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"tidyr\")\nlibrary(\"ggplot2\")\nlibrary(\"scoringutils\")\nlibrary(\"qra\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.\n\n\n\n\n\nWe set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Evaluating forecasts from multiple models"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation-of-multiple-models.html#source-file",
+ "href": "sessions/forecast-evaluation-of-multiple-models.html#source-file",
+ "title": "Evaluating forecasts from multiple models",
+ "section": "",
+ "text": "The source file of this session is located at sessions/forecast-ensembles.qmd.",
+ "crumbs": [
+ "Evaluating forecasts from multiple models"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation-of-multiple-models.html#libraries-used",
+ "href": "sessions/forecast-evaluation-of-multiple-models.html#libraries-used",
+ "title": "Evaluating forecasts from multiple models",
+ "section": "",
+ "text": "In this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and tidyr packages for data wrangling, ggplot2 library for plotting, the tidybayes package for extracting results of the inference and the scoringutils package for evaluating forecasts. We will also use qra for quantile regression averaging in the weighted ensemble section.\n\nlibrary(\"nfidd\")\nlibrary(\"dplyr\")\nlibrary(\"tidyr\")\nlibrary(\"ggplot2\")\nlibrary(\"scoringutils\")\nlibrary(\"qra\")\n\n\n\n\n\n\n\nTip\n\n\n\nThe best way to interact with the material is via the Visual Editor of RStudio.",
+ "crumbs": [
+ "Evaluating forecasts from multiple models"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation-of-multiple-models.html#initialisation",
+ "href": "sessions/forecast-evaluation-of-multiple-models.html#initialisation",
+ "title": "Evaluating forecasts from multiple models",
+ "section": "",
+ "text": "We set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.\n\nset.seed(123)",
+ "crumbs": [
+ "Evaluating forecasts from multiple models"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation-of-multiple-models.html#scoring-your-forecast",
+ "href": "sessions/forecast-evaluation-of-multiple-models.html#scoring-your-forecast",
+ "title": "Evaluating forecasts from multiple models",
+ "section": "Scoring your forecast",
+ "text": "Scoring your forecast\nAgain as in the forecasting evaluation sessions, we will score the forecasts using the scoringutils package by first converting the forecasts to the scoringutils format.\n\nsc_forecasts <- forecasts |>\n left_join(onset_df, by = \"day\") |>\n filter(!is.na(.value)) |>\n as_forecast_sample(\n forecast_unit = c(\n \"target_day\", \"horizon\", \"model\"\n ),\n observed = \"onsets\",\n predicted = \".value\",\n sample_id = \".draw\"\n )\nsc_forecasts\n\nForecast type: sample\n\n\nForecast unit:\n\n\ntarget_day, horizon, and model\n\n\n\n sample_id predicted observed target_day horizon model\n <int> <num> <int> <num> <int> <char>\n 1: 1 4 2 22 1 Random walk\n 2: 2 2 2 22 1 Random walk\n 3: 3 2 2 22 1 Random walk\n 4: 4 6 2 22 1 Random walk\n 5: 5 2 2 22 1 Random walk\n --- \n671996: 996 1 1 127 14 More mechanistic\n671997: 997 7 1 127 14 More mechanistic\n671998: 998 2 1 127 14 More mechanistic\n671999: 999 1 1 127 14 More mechanistic\n672000: 1000 5 1 127 14 More mechanistic\n\n\nEverything seems to be in order. We can now calculate some metrics as we did in the forecasting concepts session.\n\nsc_scores <- sc_forecasts |>\n score()\n\nsc_scores\n\n target_day horizon model bias dss crps\n <num> <int> <char> <num> <num> <num>\n 1: 22 1 Random walk 0.440 1.937266 0.709424\n 2: 22 2 Random walk 0.828 3.215491 1.664522\n 3: 22 3 Random walk 0.636 2.748205 1.230624\n 4: 22 4 Random walk 0.885 3.932499 2.376469\n 5: 22 5 Random walk -0.331 2.627111 1.024579\n --- \n668: 127 10 More mechanistic 0.133 1.562747 0.467048\n669: 127 11 More mechanistic 0.680 2.464343 1.158455\n670: 127 12 More mechanistic 0.843 3.233157 1.656446\n671: 127 13 More mechanistic 0.802 2.848762 1.404800\n672: 127 14 More mechanistic 0.751 2.446626 1.153386\n overprediction underprediction dispersion log_score mad ae_median\n <num> <num> <num> <num> <num> <num>\n 1: 0.240 0.000 0.469424 1.671874 1.4826 1\n 2: 1.118 0.000 0.546522 2.231108 2.9652 3\n 3: 0.656 0.000 0.574624 2.011411 2.9652 2\n 4: 1.668 0.000 0.708469 2.605082 2.9652 4\n 5: 0.000 0.238 0.786579 2.277217 2.9652 2\n --- \n668: 0.000 0.000 0.467048 1.615033 1.4826 0\n669: 0.680 0.000 0.478455 1.845716 1.4826 2\n670: 1.170 0.000 0.486446 2.282375 1.4826 2\n671: 0.996 0.000 0.408800 2.092336 1.4826 2\n672: 0.782 0.000 0.371386 1.817025 1.4826 2\n se_mean\n <num>\n 1: 1.943236\n 2: 8.479744\n 3: 5.654884\n 4: 16.378209\n 5: 1.177225\n --- \n668: 0.264196\n669: 4.313929\n670: 7.300804\n671: 5.503716\n672: 4.048144\n\n\n\nAt a glance\nLet’s summarise the scores by model first.\n\nsc_scores |>\n summarise_scores(by = \"model\")\n\n model bias dss crps overprediction\n <char> <num> <num> <num> <num>\n1: Random walk 0.21751786 6.380853 13.820203 8.750536\n2: More statistical 0.02879464 6.488302 11.974884 6.488286\n3: More mechanistic 0.23843304 5.548895 6.334783 2.061241\n underprediction dispersion log_score mad ae_median se_mean\n <num> <num> <num> <num> <num> <num>\n1: 0.717125 4.352542 3.987173 18.251203 18.607143 1595.3668\n2: 1.314455 4.172143 4.015102 17.149181 15.910714 1104.6347\n3: 2.284071 1.989470 3.704554 8.511712 8.928571 176.2964\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nBefore we look in detail at the scores, what do you think the scores are telling you? Which model do you think is best?\n\n\n\n\nContinuous ranked probability score\nAs in the forecasting evaluation session, we will start by looking at the CRPS by horizon and forecast date.\n\n\n\n\n\n\nReminder: Key things to note about the CRPS\n\n\n\n\nSmall values are better\nWhen scoring absolute values (e.g. number of cases) it can be difficult to use to compare forecasts across scales (i.e., when case numbers are different, for example between locations or at different times).\n\n\n\nFirst by forecast horizon.\n\nsc_scores |>\n summarise_scores(by = c(\"model\", \"horizon\")) |>\n ggplot(aes(x = horizon, y = crps, col = model)) +\n geom_point()\n\n\n\n\n\n\n\n\nand across different forecast dates\n\nsc_scores |>\n summarise_scores(by = c(\"target_day\", \"model\")) |>\n ggplot(aes(x = target_day, y = crps, col = model)) +\n geom_point()\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nHow do the CRPS values change based on forecast date? How do the CRPS values change with forecast horizon?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nHow do the CRPS values change based on forecast horizon?\n\nAll models have increasing CRPS with forecast horizon.\nThe more mechanistic model has the lowest CRPS at all forecast horizon.\nThe more stastical model starts to outperform the random walk model at longer time horizons.\n\nHow do the CRPS values change with forecast date?\n\nThe more statistical model does particularly poorly around the peak of the outbreak but outperforms the random walk model.\nThe more mechanistic model does particularly well around the peak of the outbreak versus all other models\nThe more statistical model starts to outperform the other models towards the end of the outbreak.\n\n\n\n\n\n\nPIT histograms\n\n\n\n\n\n\nReminder: Interpreting the PIT histogram\n\n\n\n\nIdeally PIT histograms should be uniform.\nIf is a U shape then the model is overconfident and if it is an inverted U shape then the model is underconfident.\nIf it is skewed then the model is biased towards the direction of the skew.\n\n\n\nLet’s first look at the overall PIT histogram.\n\nsc_forecasts |>\n get_pit_histogram(by = \"model\") |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n facet_wrap(~model)\n\n\n\n\n\n\n\n\nAs before let’s look at the PIT histogram by forecast horizon (to save space we will group horizons)\n\nsc_forecasts |>\n mutate(group_horizon = case_when(\n horizon <= 3 ~ \"1-3\",\n horizon <= 7 ~ \"4-7\",\n horizon <= 14 ~ \"8-14\"\n )) |>\n get_pit_histogram(by = c(\"model\", \"group_horizon\")) |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n facet_grid(vars(model), vars(group_horizon))\n\n\n\n\n\n\n\n\nand then for different forecast dates.\n\nsc_forecasts |>\n get_pit_histogram(by = c(\"model\", \"target_day\")) |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n facet_grid(vars(model), vars(target_day))\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nWhat do you think of the PIT histograms?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nWhat do you think of the PIT histograms?\n\nThe more mechanistic model is reasonably well calibrated but has a slight tendency to be overconfident.\nThe random walk is biased towards overpredicting.\nThe more statistical model is underconfident.\nAcross horizons the more mechanistic model is only liable to underpredict at the longest horizons.\nThe random walk model is initially relatively unbiased and well calibrated but becomes increasingly likely to overpredict as the horizon increases.\nThe forecast date stratified PIT histograms are hard to interpret. We may need to find other ways to visualise bias and calibration at this level of stratification (see the {scoringutils} documentation for some ideas).",
+ "crumbs": [
+ "Evaluating forecasts from multiple models"
+ ]
+ },
+ {
+ "objectID": "sessions/forecast-evaluation-of-multiple-models.html#scoring-on-the-log-scale",
+ "href": "sessions/forecast-evaluation-of-multiple-models.html#scoring-on-the-log-scale",
+ "title": "Evaluating forecasts from multiple models",
+ "section": "Scoring on the log scale",
+ "text": "Scoring on the log scale\nAgain as in the forecast evaluation session, we will also score the forecasts on the log scale.\n\nlog_sc_forecasts <- sc_forecasts |>\n transform_forecasts(\n fun = log_shift,\n offset = 1,\n append = FALSE\n )\n\nlog_sc_scores <- log_sc_forecasts |>\n score()\n\n\n\n\n\n\n\nTip\n\n\n\nReminder: For more on scoring on the log scale see the paper by @bosse2023scorin.\n\n\n\nAt a glance\n\nlog_sc_scores |>\n summarise_scores(by = \"model\")\n\n model bias dss crps overprediction\n <char> <num> <num> <num> <num>\n1: Random walk 0.175232143 -0.4690178 0.2299255 0.11532350\n2: More statistical -0.009196429 -0.4280965 0.2405370 0.08154039\n3: More mechanistic 0.195008929 -1.0961907 0.1861159 0.11374787\n underprediction dispersion log_score mad ae_median se_mean\n <num> <num> <num> <num> <num> <num>\n1: 0.02749138 0.08711065 0.5889563 0.3761217 0.3078707 0.1778818\n2: 0.05800097 0.10099568 0.6801200 0.4197095 0.3242281 0.1972211\n3: 0.02078127 0.05158679 0.6008573 0.2182572 0.2619873 0.1229412\n\n\n\n\n\n\n\n\nTake 2 minutes\n\n\n\nBefore we look in detail at the scores, what do you think the scores are telling you? Which model do you think is best?\n\n\n\n\nCRPS\n\nlog_sc_scores |>\n summarise_scores(by = c(\"model\", \"horizon\")) |>\n ggplot(aes(x = horizon, y = crps, col = model)) +\n geom_point()\n\n\n\n\n\n\n\n\n\nlog_sc_scores |>\n summarise_scores(by = c(\"target_day\", \"model\")) |>\n ggplot(aes(x = target_day, y = crps, col = model)) +\n geom_point()\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nHow do the CRPS scores on the log scale compare to the scores on the original scale?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\n\nThe performance of the mechanistic model is more variable across forecast horizon than on the natural scale.\nOn the log scale the by horizon performance of the random walk and more statistical mdoel is more comparable than on the log scale.\nThe period of high incidence dominates the target day stratified scores less on the log scale. We see that all models performed less well early and late on.\n\n\n\n\n\n\nPIT histograms\n\nlog_sc_forecasts |>\n get_pit_histogram(by = \"model\") |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n facet_wrap(~model)\n\n\n\n\n\n\n\n\n\nlog_sc_forecasts |>\n mutate(group_horizon = case_when(\n horizon <= 3 ~ \"1-3\",\n horizon <= 7 ~ \"4-7\",\n horizon <= 14 ~ \"8-14\"\n )) |>\n get_pit_histogram(by = c(\"model\", \"group_horizon\")) |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n facet_grid(vars(model), vars(group_horizon))\n\n\n\n\n\n\n\n\n\nlog_sc_forecasts |>\n get_pit_histogram(by = c(\"model\", \"target_day\")) |>\n ggplot(aes(x = mid, y = density)) +\n geom_col() +\n facet_grid(vars(model), vars(target_day))\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTake 5 minutes\n\n\n\nWhat do you think of the PIT histograms?\n\n\n\n\n\n\n\n\nSolution\n\n\n\n\n\nThe PIT histograms are similar to the original scale PIT histograms but the mechanistic model appears better calibrated.",
+ "crumbs": [
+ "Evaluating forecasts from multiple models"
+ ]
+ }
+]
\ No newline at end of file
diff --git a/sessions.html b/sessions.html
new file mode 100644
index 0000000..acd992f
--- /dev/null
+++ b/sessions.html
@@ -0,0 +1,829 @@
+
+
+
+
+
+
+
+
+
+Sessions – UEIFID
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Sessions
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Session 0: Introduction and course overview
+
13:30-13:35
+
+
Introduction to the course and the instructors (5 mins)
+
+
+
+
Session 1: Forecasting
+
13:35-14:10
+
+
Introduction to forecasting as an epidemiological problem (10 mins)
+
Practice session: Visualising infectious disease forecasts (25 minutes)
+
+
+
+
Session 2: Forecast evaluation
+
14:10-15:00
+
+
An introduction to forecast evaluation (5 mins)
+
Practice session: Evaluating forecasts from a range of models (40 mins)
+
Wrap up (5 mins)
+
+
+
+
Session 3: Evaluating forecasts from multiple models
+
15:10-15:50
+
+
Why might we want to evaluate forecasts from multiple models? (5 mins)
+
Practice session: Evaluating forecasts from multiple models (40 mins)
+
Wrap up (5 mins)
+
+
There is a coffee break at 15:30-15:40 you are welcome to attend this or keep working through the course material.
+
+
+
Session 4: Forecast ensembles
+
15:50-17:10
+
+
Introduction to forecast ensembles (10 mins)
+
Practice session: Creating forecast ensembles (60 mins)
We hope that you’ve enjoyed taking this course on understanding and evaluating forecasts of infectious disease burden. We will be very happy to have your feedback, comments or any questions on the course discussion page.
The following is a highly subjective list of papers we would recommend to read for those interested in engaging further with the topics discussed here.
As we saw in the forecast evaluation session, different modelling approaches have different strength and weaknesses, and it is not clear a priori which one produces the best forecast in any given situation. One way to attempt to draw strength from a diversity of approaches is the creation of so-called forecast ensembles from the forecasts produced by different models.
+
In this session, we’ll build ensembles using forecasts from models of different levels of mechanism vs. statistical complexity. We will then compare the performance of these ensembles to the individual models and to each other. Rather than using the forecast samples we have been using we will instead now use quantile-based forecasts.
+
+
+
+
+
+
+Representations of probabilistic forecasts
+
+
+
+
+
+
Probabilistic predictions can be described as coming from a probabilistic probability distributions. In general and when using complex models such as the one we discuss in this course, these distributions can not be expressed in a simple analytical formal as we can do if, e.g. talking about common probability distributions such as the normal or gamma distributions. Instead, we typically use a limited number of samples generated from Monte-Carlo methods to represent the predictive distribution. However, this is not the only way to characterise distributions.
+
A quantile is the value that corresponds to a given quantile level of a distribution. For example, the median is the 50th quantile of a distribution, meaning that 50% of the values in the distribution are less than the median and 50% are greater. Similarly, the 90th quantile is the value that corresponds to 90% of the distribution being less than this value. If we characterise a predictive distribution by its quantiles, we specify these values at a range of specific quantile levels, e.g. from 5% to 95% in 5% steps.
+
Deciding how to represent forecasts depends on many things, for example the method used (and whether it produces samples by default) but also logistic considerations. Many collaborative forecasting projects and so-called forecasting hubs use quantile-based representations of forecasts in the hope to be able to characterise both the centre and tails of the distributions more reliably and with less demand on storage space than a sample-based representation.
The aim of this session is to introduce the concept of ensembles of forecasts and to evaluate the performance of ensembles of the multiple models.
+
+
+
+
+
+
+Setup
+
+
+
+
+
+
+
Source file
+
The source file of this session is located at sessions/forecast-ensembles.qmd.
+
+
+
Libraries used
+
In this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and tidyr packages for data wrangling, ggplot2 library for plotting, the tidybayes package for extracting results of the inference and the scoringutils package for evaluating forecasts. We will also use qrensemble for quantile regression averaging in the weighted ensemble section.
The best way to interact with the material is via the Visual Editor of RStudio.
+
+
+
+
+
Initialisation
+
We set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.
+
+
set.seed(123)
+
+
+
+
+
+
+
+
+
Individual forecast models
+
In this session we will use the forecasts from different models we introduced in the forecasting evaluation of multiple models session. There all shared the same basic renewal with delays structure but used different models for the evolution of the effective reproduction number over time. These were:
+
+
A random walk model
+
A simple model of susceptible depletion referred to as “More mechanistic”
+
A differenced autoregressive model referred to as “More statistical”
+
+
As previously, we have fitted these models to a range of forecast dates so you don’t have to wait for the models to fit.
+
+
data(rw_forecasts, stat_forecasts, mech_forecasts)
+forecasts <-bind_rows(
+ rw_forecasts,
+mutate(stat_forecasts, model ="More statistical"),
+mutate(mech_forecasts, model ="More mechanistic")
+) |>
+ungroup()
+
+forecasts
+
+
# A tibble: 672,000 × 7
+ day .draw .variable .value horizon target_day model
+ <dbl> <int> <chr> <dbl> <int> <dbl> <chr>
+ 1 23 1 forecast 4 1 22 Random walk
+ 2 23 2 forecast 2 1 22 Random walk
+ 3 23 3 forecast 2 1 22 Random walk
+ 4 23 4 forecast 6 1 22 Random walk
+ 5 23 5 forecast 2 1 22 Random walk
+ 6 23 6 forecast 3 1 22 Random walk
+ 7 23 7 forecast 5 1 22 Random walk
+ 8 23 8 forecast 2 1 22 Random walk
+ 9 23 9 forecast 3 1 22 Random walk
+10 23 10 forecast 5 1 22 Random walk
+# ℹ 671,990 more rows
+
+
+
+
+
+
+
+
+How did we generate these forecasts?
+
+
+
+
+
+
Some important things to note about these forecasts:
+
+
We used a 14 day forecast horizon.
+
Each forecast used all the data up to the forecast date.
+
We generated 1000 predictive posterior samples for each forecast.
+
We started forecasting 3 weeks into the outbreak and then forecast every 7 days until the end of the data (excluding the last 14 days to allow a full forecast).
+
We use the same simulated outbreak data as before:
Converting sample-based forecasts to quantile-based forecasts
+
As in this session we will be thinking about forecasts in terms quantiles of the predictive distributions, we will need to convert our sample based forecasts to quantile-based forecasts. We will do this by focusing on the marginal distribution at each predicted time point, that is we treat each time point as independent of all others and calculate quantiles based on the sample predictive trajectories at that time point. An easy way to do this is to use the {scoringutils} package. The steps to do this are to first declare the forecasts as sample forecasts.
+ observed target_day horizon model day quantile_level
+ <int> <num> <int> <char> <num> <num>
+ 1: 4 22 1 Random walk 23 0.05
+ 2: 4 22 1 Random walk 23 0.25
+ 3: 4 22 1 Random walk 23 0.50
+ 4: 4 22 1 Random walk 23 0.75
+ 5: 4 22 1 Random walk 23 0.95
+ ---
+3356: 2 127 14 More mechanistic 141 0.05
+3357: 2 127 14 More mechanistic 141 0.25
+3358: 2 127 14 More mechanistic 141 0.50
+3359: 2 127 14 More mechanistic 141 0.75
+3360: 2 127 14 More mechanistic 141 0.95
+ predicted
+ <num>
+ 1: 0
+ 2: 2
+ 3: 3
+ 4: 5
+ 5: 7
+ ---
+3356: 1
+3357: 2
+3358: 3
+3359: 4
+3360: 6
+
+
+
+
+
+
+
+
+What is happening here?
+
+
+
+
+
+
+
Internally scoringutils is calculating the quantiles of the sample-based forecasts.
+
It does this by using a set of default quantiles but different ones can be specified by the user to override the default.
+
It then calls the quantile() function from base R to calculate the quantiles.
+
This is estimating the value that corresponds to each given quantile level by ordering the samples and then taking the value at the appropriate position.
+
+
+
+
+
+
+
Simple unweighted ensembles
+
A good place to start when building ensembles is to take the mean or median of the unweighted forecast at each quantile level, and treat these as quantiles of the ensemble predictive distribution. Typically, the median is preferred when outlier forecasts are likely to be present as it is less sensitive to these. However, the mean is preferred when forecasters have more faith in models that diverge from the median performance and want to represent this in the ensemble.
+
+
+
+
+
+
+Vincent average
+
+
+
+
+
+
The procedure of calculating quantiles of a new distribution as a weighted average of quantiles of constituent distributions (e.g., different measurements) is called a Vincent average, after the biologist Stella Vincent who described this as early as 1912 when studying the function of whiskers in the behaviour of white rats.
+
+
+
+
+
Construction
+
We can calculate the mean ensemble by taking the mean of the forecasts at each quantile level.
How do these ensembles compare to the individual models?
+
How do they differ from each other?
+
Are there any differences across forecast dates?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
How do these ensembles compare to the individual models?
+
+
As before, the ensembles appear to be less variable than the statistical models but more variable than the mechanistic model.
+
+
How do they differ from each other?
+
+
The mean ensemble has marginally tighter uncertainty bounds than the median ensemble as for the single forecast.
+
+
Are there any differences across forecast dates?
+
+
The mean ensemble appears to be more variable across forecast dates than the median ensemble with this being more pronounced after the peak of the outbreak.
+
+
+
+
+
+
+
Evaluation
+
As in the forecast evaluation session, we can evaluate the accuracy of the ensembles using the {scoringutils} package and in particular the score() function.
The weighted interval score (WIS) is a proper scoring rule for quantile forecasts that approximates the Continuous Ranked Probability Score (CRPS) by considering a weighted sum of multiple prediction intervals. As the number of intervals increases, the WIS converges to the CRPS, combining sharpness and penalties for over- and under-prediction.
+
We see it here as we are scoring quantiles and not samples hence we cannot use CRPS as we did before.
+
+
+
Again we start with a high level overview of the scores by model.
What do you think the scores are telling you? Which model do you think is best? What other scoring breakdowns might you want to look at?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
What do you think the scores are telling you? Which model do you think is best?
+
+
The mean ensemble appears to be the best performing ensemble model overall.
+
However, the more mechanistic model appears to be the best performing model overall.
+
+
What other scoring breakdowns might you want to look at?
+
+
There might be variation over forecast dates or horizons between the different ensemble methods
+
+
+
+
+
+
+
+
Unweighted ensembles of filtered models
+
A simple method that is often used to improve ensemble performance is to prune out models that perform very poorly. Balancing this can be tricky however as it can be hard to know how much to prune. The key tradeoff to consider is how much to optimise for which models have performed well in the past (and what your definition of the past is, for example all time or only the last few weeks) versus how much you want to allow for the possibility that these models may not perform well in the future.
+
+
Construction
+
As we just saw, the random walk model (our original baseline model) is performing poorly in comparison to the other models. We can remove this model from the ensemble and see if this improves the performance of the ensemble.
+
+
+
+
+
+
+Warning
+
+
+
+
Here we are technically cheating a little as we are using the test data to help select the models to include in the ensemble. In the real world you would not do this as you would not have access to the test data and so this is an idealised scenario.
How do the filtered ensembles compare to the simple ensembles?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
How do the filtered ensembles compare to the simple ensembles?
+
+
The filtered ensembles appear to be more accurate than the simple ensembles.
+
As you would expect they are an average of the more mechanistic model and the more statistical model.
+
As there are only two models in the ensemble, the median and mean ensembles are identical.
+
For the first time there are features of the ensemble that outperform the more mechanistic model though it remains the best performing model overall.
+
+
+
+
+
+
+
+
Weighted quantile ensembles
+
The simple mean and median we used to average quantiles earlier treats every model as the same. We could try to improve performance by replacing this with a weighted mean (or weighted median), for example given greater weight to models that have proven to make better forecasts. Here we will explore two common weighting methods:
+
+
Inverse WIS weighting
+
Quantile regression averaging
+
+
Inverse WIS weighting is a simple method that weights the forecasts by the inverse of their WIS over some period (note that identifying what this period should be in order to produce the best forecasts is not straightforward as predictive performance may vary over time if models are good at different things). The main benefit of WIS weighting over other methods is that it is simple to understand and implement. However, it does not optimise the weights directly to produce the best forecasts. It relies on the hope that giving more weight to better performing models yields a better ensemble
+
Quantile regression averaging on the other hand does optimise the weights directly in order to yield the best scores on past data.
+
+
Construction
+
+
Inverse WIS weighting
+
In order to perform inverse WIS weighting we first need to calculate the WIS for each model. We already have this from the previous evaluation so we can reuse this.
Now lets apply the weights to the forecast models. As we can only use information that was available at the time of the forecast to perform the weighting, we use weights from two weeks prior to the forecast date to inform each ensemble.
We futher to perform quantile regression averaging (QRA) for each forecast date. Again we need to consider how many previous forecasts we wish to use to inform each ensemble forecast. Here we decide to use up to 3 weeks of previous forecasts to inform each QRA ensemble.
Instead of creating a single optimised ensemble and using this for all forecast horizons we might also want to consider a separate optimised QRA ensemble for each forecast horizon, reflecting that models might perform differently depending on how far ahead a forecast is produced. We can do this using qra() with the group argument.
How do the weighted ensembles compare to the simple ensembles? Which do you think is best? Are you surprised by the results? Can you think of any reasons that would explain them?
+
+
+
+
+
Models weights
+
Now that we have a weighted ensemble, we can also look at the weights of the individual models. Here we do this for the quantile regression averaging ensemble but we could also do this for the inverse WIS ensemble or any other weighted ensemble (for an unweighted ensemble the weights are all equal).
+
+
qra_weights |>
+ggplot(aes(x = target_day, y = weight, fill = model)) +
+geom_col(position ="stack") +
+theme(legend.position ="bottom")
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Take 2 minutes
+
+
+
+
How do the weights change over time? Are you surprised by the results given what you know about the models performance?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
How do the weights change over time?
+
+
Early on the more statistical models have higher weights
+
Gradually the random walk model gains weight and by the end of the forecast horizon it represents the entire ensemble.
+
Near the peak the mechanistic model also gains weight.
+
+
Are you surprised by the results given what you know about the models performance?
+
+
As the random walk model is performing poorly, you would expect it to have low weights but actually it often doesn’t. This implies that its poor performance is restricted to certain parts of the outbreak.
+
Even though the mechanistic model performs really well overall it is only included in the ensemble around the peak. This could be because the training data doesn’t include changes in the epidemic dynamics and so the mechanistic model is not given sufficient weight.
+
+
+
+
+
+
+
+
Evaluation
+
For a final evaluation we can look at the scores for each model and ensemble again. We remove the two weeks of forecasts as these do not have a quantile regression average forecasts as these require training data to estimate.
How do the weighted ensembles compare to the simple ensembles on the natural and log scale?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
The best ensembles slightly outperform some of the simple ensembles but there is no obvious benefit from using weighted ensembles. Why might this be the case?
+
+
+
+
+
+
+
Wrap up
+
+
+
References
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-model-weights-1.png b/sessions/forecast-ensembles_files/figure-html/plot-model-weights-1.png
new file mode 100644
index 0000000..e5ae3a7
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-model-weights-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-multiple-filtered-forecasts-1.png b/sessions/forecast-ensembles_files/figure-html/plot-multiple-filtered-forecasts-1.png
new file mode 100644
index 0000000..d52934e
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-multiple-filtered-forecasts-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-multiple-filtered-forecasts-log-1.png b/sessions/forecast-ensembles_files/figure-html/plot-multiple-filtered-forecasts-log-1.png
new file mode 100644
index 0000000..abe85e5
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-multiple-filtered-forecasts-log-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-multiple-forecasts-1.png b/sessions/forecast-ensembles_files/figure-html/plot-multiple-forecasts-1.png
new file mode 100644
index 0000000..49a7e62
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-multiple-forecasts-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-multiple-forecasts-log-1.png b/sessions/forecast-ensembles_files/figure-html/plot-multiple-forecasts-log-1.png
new file mode 100644
index 0000000..384af11
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-multiple-forecasts-log-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-multiple-weighted-forecasts-1.png b/sessions/forecast-ensembles_files/figure-html/plot-multiple-weighted-forecasts-1.png
new file mode 100644
index 0000000..aa82290
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-multiple-weighted-forecasts-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-multiple-weighted-forecasts-log-1.png b/sessions/forecast-ensembles_files/figure-html/plot-multiple-weighted-forecasts-log-1.png
new file mode 100644
index 0000000..53968d2
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-multiple-weighted-forecasts-log-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-single-filtered-forecasts-1.png b/sessions/forecast-ensembles_files/figure-html/plot-single-filtered-forecasts-1.png
new file mode 100644
index 0000000..81cf1db
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-single-filtered-forecasts-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-single-filtered-forecasts-log-1.png b/sessions/forecast-ensembles_files/figure-html/plot-single-filtered-forecasts-log-1.png
new file mode 100644
index 0000000..250320d
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-single-filtered-forecasts-log-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-single-forecasts-1.png b/sessions/forecast-ensembles_files/figure-html/plot-single-forecasts-1.png
new file mode 100644
index 0000000..a019378
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-single-forecasts-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-single-forecasts-log-1.png b/sessions/forecast-ensembles_files/figure-html/plot-single-forecasts-log-1.png
new file mode 100644
index 0000000..7a7f64f
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-single-forecasts-log-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-single-weighted-forecasts-1.png b/sessions/forecast-ensembles_files/figure-html/plot-single-weighted-forecasts-1.png
new file mode 100644
index 0000000..468709d
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-single-weighted-forecasts-1.png differ
diff --git a/sessions/forecast-ensembles_files/figure-html/plot-single-weighted-forecasts-log-1.png b/sessions/forecast-ensembles_files/figure-html/plot-single-weighted-forecasts-log-1.png
new file mode 100644
index 0000000..94d626b
Binary files /dev/null and b/sessions/forecast-ensembles_files/figure-html/plot-single-weighted-forecasts-log-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models.html b/sessions/forecast-evaluation-of-multiple-models.html
new file mode 100644
index 0000000..b931024
--- /dev/null
+++ b/sessions/forecast-evaluation-of-multiple-models.html
@@ -0,0 +1,1921 @@
+
+
+
+
+
+
+
+
+
+Evaluating forecasts from multiple models – UEIFID
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Evaluating forecasts from multiple models
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Introduction
+
We can classify models along a spectrum by how much they include an understanding of underlying processes, or mechanisms; or whether they emphasise drawing from the data using a statistical approach. These different approaches all have different strength and weaknesses, and it is not clear a prior which one produces the best forecast in any given situation.
+
In this session, we’ll start with forecasts from models of different levels of mechanism vs. statistical complexity and evaluate them using visualisation and proper scoring rules as we did in the last session for the random walk model
The aim of this session is to introduce the concept of a spectrum of forecasting models and to demonstrate how to evaluate a range of different models from across this spectrum.
+
+
+
+
+
+
+Setup
+
+
+
+
+
+
+
Source file
+
The source file of this session is located at sessions/forecast-ensembles.qmd.
+
+
+
Libraries used
+
In this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and tidyr packages for data wrangling, ggplot2 library for plotting, the tidybayes package for extracting results of the inference and the scoringutils package for evaluating forecasts. We will also use qra for quantile regression averaging in the weighted ensemble section.
The best way to interact with the material is via the Visual Editor of RStudio.
+
+
+
+
+
Initialisation
+
We set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.
+
+
set.seed(123)
+
+
+
+
+
+
+
+
+
Individual forecast models
+
In this session we will use the forecasts from different models. There all shared the same basic renewal with delays structure but used different models for the evolution of the effective reproduction number over time. These were:
+
+
A random walk model (what we have looked at so far)
+
A simple model of susceptible depletion referred to as “More mechanistic”
+
A differenced autoregressive model referred to as “More statistical”
+
+
For the purposes of this session the precise details of the models are not critical to the concepts we are exploring.
+
+
+
+
+
+
+More information on the mechanistic model (optional)
+
+
+
+
+
+
One way to potentially improve the renewal model is to add more mechanistic structure. In the forecasting visualisation session, we saw that the renewal model was making unbiased forecasts when the reproduction number was constant but that it overestimated the number of cases when the reproduction number was reducing due to susceptible depletion.
+
+
+
+
+
+
+Warning
+
+
+
+
This is slightly cheating as we know the future of this outbreak and so can make a model to match. This is easy to do and important to watch for if wanting to make generalisable methods.
+
+
+
This suggests that we should add a term to the renewal model which captures the depletion of susceptibles. One way to do this is to add a term which is proportional to the number of susceptibles in the population. This is the idea behind the SIR model which is a simple compartmental model of infectious disease transmission. If we assume that susceptible depletion is the only mechanism which is causing the reproduction number to change, we can write the reproduction model as:
+
\[
+R_t = \frac{S_{t-1}}{N} R_0
+\]
+
+
+
+
+
+
+Note
+
+
+
+
This approximates susceptible depletion as a linear function of the number of susceptibles in the population. This is a simplification but it is a good starting point.
+
+
+
+
+
+
+
+
+What behaviour would we expect from this model?
+
+
+
+
+
+
+
n <-100
+N <-1000
+R0 <-1.5
+S <-rep(NA, n)
+S[1] <- N
+Rt <-rep(NA, n) ## reproduction number
+Rt[1] <- R0
+I <-rep(NA, n)
+I[1] <-1
+for (i in2:n) {
+ Rt[i] <- (S[i-1]) / N * R0
+ I[i] <- I[i-1] * Rt[i]
+ S[i] <- S[i-1] - I[i]
+}
+
+data <-tibble(t =1:n, Rt = Rt)
+
+ggplot(data, aes(x = t, y = Rt)) +
+geom_line() +
+labs(title ="Simulated data from an SIR model",
+x ="Time",
+y ="Rt")
+
+
+
+
+
+
+
+
+
+
+
+
The key assumptions we are making here are:
+
+
The population is constant and we roughly know the size of the population.
+
The reproduction number only changes due to susceptible depletion
+
The number of new cases at each time is proportional to the number of susceptibles in the population.
+
+
+
+
+
+
+
+
+
+
+More information on the statistical model (optional)
+
+
+
+
+
+
Adding more mechanistic structure is not always possible and, if we don’t specify mechanisms correctly, might make forecasts worse. Rather than adding more mechanistic structure to the renewal model, we could add more statistical structure with the aim of improving performance. Before we do this, we need to think about what we want from a forecasting model. As we identified above, we want a model which is unbiased and which has good short-term forecasting properties. We know that we want it to be able to adapt to trends in the reproduction number and that we want it to be able to capture the noise in the data. A statistical term that can be used to describe a time series with a trend is saying that the time series is non-stationary. More specifically, a stationary time series is defined as one whose statistical properties, such as mean and variance, do not change over time. In infectious disease epidemiology, this would only be expected for endemic diseases without external seasonal influence.
+
The random walk model we used in the forecasting visualisation session is a special case of a more general class of models called autoregressive (AR) models. AR models are a class of models which predict the next value in a time series as a linear combination of the previous values in the time series. The random walk model is specifically a special case of an AR(1) model where the next value in the time series is predicted as the previous value, multiplied by a value between 1 and -1 , plus some noise. This becomes a random walk when the multipled value is 0.
+
For the log-transformed reproduction number (\(log(R_t)\)), the model is:
+
\[
+log(R_t) = \phi log(R_{t-1}) + \epsilon_t
+\]
+
where \(\epsilon_t\) is a normally distributed error term with mean 0 and standard deviation \(\sigma\) and \(\phi\) is a parameter between -1 and 1. If we restrict \(\phi\) to be between 0 and 1, we get a model which is biased towards a reproduction number of 1 but which can still capture trends in the data that decay over time.
+
+
+
+
+
+
+What behaviour would we expect from this model?
+
+
+
+
+
+
+
n <-100
+phi <-0.4
+sigma <-0.1
+log_R <-rep(NA, n)
+log_R[1] <-rnorm(1, 0, sigma)
+for (i in2:n) {
+ log_R[i] <- phi * log_R[i-1] +rnorm(1, 0, sigma)
+}
+data <-tibble(t =1:n, R =exp(log_R))
+
+ggplot(data, aes(x = t, y = R)) +
+geom_line() +
+labs(title ="Simulated data from an exponentiated AR(1) model",
+x ="Time",
+y ="R")
+
+
+
+
+
+
+
+
+
+
+
+
However, we probably don’t want a model which is biased towards a reproduction number of 1 (unless we have good reason to believe that is the expected behaviour). So what should we do?
+
Returning to the idea that the reproduction number is a non-stationary time series, as we expect to have a trend in the reproduction numbers we want to capture, we can use a method from the field of time series analysis called differencing to make the time series stationary. This involves taking the difference between consecutive values in the time series. For the log-transformed reproduction number, this would be:
We can use this function to simulate a differenced AR process:
+
+
log_R <-geometric_diff_ar(init =1, noise =rnorm(100), std =0.1, damp =0.1)
+
+data <-tibble(t =seq_along(log_R), R =exp(log_R))
+
+ggplot(data, aes(x = t, y = R)) +
+geom_line() +
+labs(title ="Simulated data from an exponentiated AR(1) model with differencing",
+x ="Time",
+y ="R")
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
As previously, we have fitted these models to a range of forecast dates so you don’t have to wait for the models to fit. We will now evaluate the forecasts from the mechanistic and statistical models.
+
+
data(rw_forecasts, stat_forecasts, mech_forecasts)
+forecasts <-bind_rows(
+ rw_forecasts,
+mutate(stat_forecasts, model ="More statistical"),
+mutate(mech_forecasts, model ="More mechanistic")
+) |>
+ungroup()
+
+forecasts
+
+
# A tibble: 672,000 × 7
+ day .draw .variable .value horizon target_day model
+ <dbl> <int> <chr> <dbl> <int> <dbl> <chr>
+ 1 23 1 forecast 4 1 22 Random walk
+ 2 23 2 forecast 2 1 22 Random walk
+ 3 23 3 forecast 2 1 22 Random walk
+ 4 23 4 forecast 6 1 22 Random walk
+ 5 23 5 forecast 2 1 22 Random walk
+ 6 23 6 forecast 3 1 22 Random walk
+ 7 23 7 forecast 5 1 22 Random walk
+ 8 23 8 forecast 2 1 22 Random walk
+ 9 23 9 forecast 3 1 22 Random walk
+10 23 10 forecast 5 1 22 Random walk
+# ℹ 671,990 more rows
+
+
+
+
+
+
+
+
+How did we generate these forecasts?
+
+
+
+
+
+
Some important things to note about these forecasts:
+
+
We used a 14 day forecast horizon.
+
Each forecast used all the data up to the forecast date.
+
We generated 1000 predictive posterior samples for each forecast.
+
We started forecasting 3 weeks into the outbreak and then forecast every 7 days until the end of the data (excluding the last 14 days to allow a full forecast).
+
We use the same simulated outbreak data as before:
How do these forecasts compare? Which do you prefer?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
How do these forecasts compare?
+
+
The more mechanistic model captures the downturn in the data very well.
+
Past the peak all models were comparable.
+
The more statistical model captures the downturn faster than the random walk but less fast than the more mechanistic mode.
+
The more statistical model sporadically predicts a more rapidly growing outbreak than occurred early on.
+
The more statistical model is more uncertain than the mechanistic model but less uncertain than the random walk.
+
+
Which do you prefer?
+
+
The more mechanistic model seems to be the best at capturing the downturn in the data and the uncertainty in the forecasts seems reasonable.
+
If we weren’t confident in the effective susceptible population the AR model might be preferable.
+
+
+
+
+
+
Scoring your forecast
+
Again as in the forecasting evaluation sessions, we will score the forecasts using the scoringutils package by first converting the forecasts to the scoringutils format.
Before we look in detail at the scores, what do you think the scores are telling you? Which model do you think is best?
+
+
+
+
+
Continuous ranked probability score
+
As in the forecasting evaluation session, we will start by looking at the CRPS by horizon and forecast date.
+
+
+
+
+
+
+Reminder: Key things to note about the CRPS
+
+
+
+
+
Small values are better
+
When scoring absolute values (e.g. number of cases) it can be difficult to use to compare forecasts across scales (i.e., when case numbers are different, for example between locations or at different times).
+
+
+
+
First by forecast horizon.
+
+
sc_scores |>
+summarise_scores(by =c("model", "horizon")) |>
+ggplot(aes(x = horizon, y = crps, col = model)) +
+geom_point()
+
+
+
+
+
+
+
+
+
and across different forecast dates
+
+
sc_scores |>
+summarise_scores(by =c("target_day", "model")) |>
+ggplot(aes(x = target_day, y = crps, col = model)) +
+geom_point()
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Take 5 minutes
+
+
+
+
How do the CRPS values change based on forecast date? How do the CRPS values change with forecast horizon?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
How do the CRPS values change based on forecast horizon?
+
+
All models have increasing CRPS with forecast horizon.
+
The more mechanistic model has the lowest CRPS at all forecast horizon.
+
The more stastical model starts to outperform the random walk model at longer time horizons.
+
+
How do the CRPS values change with forecast date?
+
+
The more statistical model does particularly poorly around the peak of the outbreak but outperforms the random walk model.
+
The more mechanistic model does particularly well around the peak of the outbreak versus all other models
+
The more statistical model starts to outperform the other models towards the end of the outbreak.
+
+
+
+
+
+
+
PIT histograms
+
+
+
+
+
+
+Reminder: Interpreting the PIT histogram
+
+
+
+
+
Ideally PIT histograms should be uniform.
+
If is a U shape then the model is overconfident and if it is an inverted U shape then the model is underconfident.
+
If it is skewed then the model is biased towards the direction of the skew.
The more mechanistic model is reasonably well calibrated but has a slight tendency to be overconfident.
+
The random walk is biased towards overpredicting.
+
The more statistical model is underconfident.
+
Across horizons the more mechanistic model is only liable to underpredict at the longest horizons.
+
The random walk model is initially relatively unbiased and well calibrated but becomes increasingly likely to overpredict as the horizon increases.
+
The forecast date stratified PIT histograms are hard to interpret. We may need to find other ways to visualise bias and calibration at this level of stratification (see the {scoringutils} documentation for some ideas).
Before we look in detail at the scores, what do you think the scores are telling you? Which model do you think is best?
+
+
+
+
+
CRPS
+
+
log_sc_scores |>
+summarise_scores(by =c("model", "horizon")) |>
+ggplot(aes(x = horizon, y = crps, col = model)) +
+geom_point()
+
+
+
+
+
+
+
+
+
+
log_sc_scores |>
+summarise_scores(by =c("target_day", "model")) |>
+ggplot(aes(x = target_day, y = crps, col = model)) +
+geom_point()
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Take 5 minutes
+
+
+
+
How do the CRPS scores on the log scale compare to the scores on the original scale?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
+
The performance of the mechanistic model is more variable across forecast horizon than on the natural scale.
+
On the log scale the by horizon performance of the random walk and more statistical mdoel is more comparable than on the log scale.
+
The period of high incidence dominates the target day stratified scores less on the log scale. We see that all models performed less well early and late on.
The PIT histograms are similar to the original scale PIT histograms but the mechanistic model appears better calibrated.
+
+
+
+
+
+
+
+
Going further
+
+
We have only looked at three forecasting models here. There are many more models that could be used. For example, we could use a more complex mechanistic model which captures more of the underlying dynamics of the data generating process. We could also use a more complex statistical model which captures more of the underlying structure of the data.
+
We could also combine the more mechanistic and more statistical models to create a hybrid model which captures the best of both worlds (maybe).
+
We could also use a more complex scoring rule to evaluate the forecasts. For example, we could use a multivariate scoring rule which captures more of the structure of the data.
+
+
+
+
Wrap up
+
+
+
References
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-1.png
new file mode 100644
index 0000000..90d5852
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-date-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-date-1.png
new file mode 100644
index 0000000..a93f784
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-date-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-date-log-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-date-log-1.png
new file mode 100644
index 0000000..aa9628a
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-date-log-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-horizon-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-horizon-1.png
new file mode 100644
index 0000000..f799387
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-horizon-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-horizon-log-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-horizon-log-1.png
new file mode 100644
index 0000000..f8492eb
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-horizon-log-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-log-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-log-1.png
new file mode 100644
index 0000000..bfc0959
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/pit-histogram-log-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/plot-all-forecasts-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/plot-all-forecasts-1.png
new file mode 100644
index 0000000..e74175f
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/plot-all-forecasts-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/plot-all-forecasts-log-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/plot-all-forecasts-log-1.png
new file mode 100644
index 0000000..cb692eb
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/plot-all-forecasts-log-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-10-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-10-1.png
new file mode 100644
index 0000000..1c7ee52
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-10-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-11-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-11-1.png
new file mode 100644
index 0000000..bf94c7f
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-11-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-2-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-2-1.png
new file mode 100644
index 0000000..ccb8e18
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-2-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-3-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-3-1.png
new file mode 100644
index 0000000..9c1fbfc
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-3-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-4-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-4-1.png
new file mode 100644
index 0000000..3f60712
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-4-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-7-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-7-1.png
new file mode 100644
index 0000000..32d062a
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-7-1.png differ
diff --git a/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-8-1.png b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-8-1.png
new file mode 100644
index 0000000..8b86252
Binary files /dev/null and b/sessions/forecast-evaluation-of-multiple-models_files/figure-html/unnamed-chunk-8-1.png differ
diff --git a/sessions/forecast-evaluation.html b/sessions/forecast-evaluation.html
new file mode 100644
index 0000000..c4b799e
--- /dev/null
+++ b/sessions/forecast-evaluation.html
@@ -0,0 +1,1746 @@
+
+
+
+
+
+
+
+
+
+Forecast evaluation – UEIFID
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Forecast evaluation
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Introduction
+
So far we have focused on visualising forecasts, including confronting them with events that were observed after the forecast was made. Besides visualising the forecasts, we can also summarise performance quantitatively. In this session you will get to know several ways of assessing different aspects of forecast performance.
The aim of this session is to introduce the concept of forecast evaluation using scoring rules.
+
+
+
+
+
+
+Setup
+
+
+
+
+
+
+
Source file
+
The source file of this session is located at sessions/forecast-evaluation.qmd.
+
+
+
Libraries used
+
In this session we will use the nfidd package to load a data set of infection times and access stan models and helper functions, the dplyr and package for data wrangling, ggplot2 library for plotting and the scoringutils package for evaluating forecasts.
The best way to interact with the material is via the Visual Editor of RStudio.
+
+
+
+
+
Initialisation
+
We set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.
+
+
set.seed(123)
+
+
+
+
+
+
+
+
+
Introduction to forecast evaluation
+
An important aspect of making forecasts is that we can later confront the forecasts with what really happened and use this to assess whether our forecast model makes good predictions, or which of multiple models work best in which situation.
+
+
+
+
+
+
+What do we look for in a good forecast? Some food for thought:
+
+
+
+
+
Calibration: The forecast should be well calibrated. This means that the forecasted probabilities should match the observed frequencies. For example, if the model predicts a 50% probability of an event occurring, then the event should occur approximately 50% of the time.
+
Unbiasedness: The forecast should be unbiased. This means that the average forecasted value should be equal to the average observed value. It shouldn’t consistently over- or underpredict.
+
Accuracy: The forecast should be accurate. This means that the forecasted values should be close to the observed values.
+
Sharpness: As long as the other conditions are fulfilled we want prediction intervals to be as narrow as possible. Predicting that “anything can happen” might be correct but not very useful.
+
+
+
+
+
+
Evaluate a forecasting model
+
We consider forecasts from the model we visualised previously, and explore different metrics for evaluation.
+
+
data(rw_forecasts)
+rw_forecasts |>
+ungroup()
+
+
# A tibble: 224,000 × 7
+ day .draw .variable .value horizon target_day model
+ <dbl> <int> <chr> <dbl> <int> <dbl> <chr>
+ 1 23 1 forecast 4 1 22 Random walk
+ 2 23 2 forecast 2 1 22 Random walk
+ 3 23 3 forecast 2 1 22 Random walk
+ 4 23 4 forecast 6 1 22 Random walk
+ 5 23 5 forecast 2 1 22 Random walk
+ 6 23 6 forecast 3 1 22 Random walk
+ 7 23 7 forecast 5 1 22 Random walk
+ 8 23 8 forecast 2 1 22 Random walk
+ 9 23 9 forecast 3 1 22 Random walk
+10 23 10 forecast 5 1 22 Random walk
+# ℹ 223,990 more rows
+
+
+
+
+
+
+
+
+How did we generate these forecasts?
+
+
+
+
+
+
Some important things to note about these forecasts:
+
+
We used a 14 day forecast horizon.
+
Each forecast used all the data up to the forecast date.
+
We generated 1000 predictive posterior samples for each forecast.
+
We started forecasting 3 weeks into the outbreak and then forecast every 7 days until the end of the data (excluding the last 14 days to allow a full forecast).
+
We use the same simulated outbreak data as before:
We now summarise performance quantitatively by using scoring metrics. Whilst some of these metrics are more useful for comparing models, many can be also be useful for understanding the performance of a single model.
+
+
+
+
+
+
+Tip
+
+
+
+
In this session, we’ll use “proper” scoring rules: these are scoring rules that make sure no model can get better scores than the true model, i.e. the model used to generate the data. Of course we usually don’t know this (as we don’t know the “true model” for real-world data) but proper scoring rules incentivise forecasters to make their best attempt at reproducing its behaviour. For a comprehensive text on proper scoring rules and their mathematical properties, we recommend the classic paper by Gneiting and Raftery (2007).
+
+
+
We will use the {scoringutils} package to calculate these metrics. Our first step is to convert our forecasts into a format that the {scoringutils} package can use. We will use as_forecast_sample() to do this:
+ sample_id predicted observed target_day horizon model
+ <int> <num> <int> <num> <int> <char>
+ 1: 1 4 4 22 1 Random walk
+ 2: 2 2 4 22 1 Random walk
+ 3: 3 2 4 22 1 Random walk
+ 4: 4 6 4 22 1 Random walk
+ 5: 5 2 4 22 1 Random walk
+ ---
+223996: 996 1 2 127 14 Random walk
+223997: 997 0 2 127 14 Random walk
+223998: 998 0 2 127 14 Random walk
+223999: 999 1 2 127 14 Random walk
+224000: 1000 0 2 127 14 Random walk
+
+
+
As you can see this has created a forecast object which has a print method that summarises the forecasts.
+
+
+
+
+
+
+Take 2 minutes
+
+
+
+
What important information is in the forecast object?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
+
The forecast unit which is the target day, horizon, and model
+
The type of forecast which is a sample forecast
+
+
+
+
+
Everything seems to be in order. We can now use the scoringutils package to calculate some metrics. We will use the default sample metrics (as our forecasts are in sample format) and score our forecasts.
target_day horizon model bias dss crps overprediction
+ <num> <int> <char> <num> <num> <num> <num>
+ 1: 22 1 Random walk -0.304 1.587880 0.623424 0.000
+ 2: 22 2 Random walk 0.828 3.215491 1.664522 1.118
+ 3: 22 3 Random walk 0.002 1.936078 0.574624 0.000
+ 4: 22 4 Random walk 0.687 3.231981 1.570469 0.862
+ 5: 22 5 Random walk 0.767 3.750147 2.172579 1.386
+ ---
+220: 127 10 Random walk -0.821 3.120249 1.745842 0.000
+221: 127 11 Random walk -0.867 3.853982 1.972866 0.000
+222: 127 12 Random walk -0.949 7.803330 3.092738 0.000
+223: 127 13 Random walk -0.914 5.003912 2.318689 0.000
+224: 127 14 Random walk -0.637 1.140452 0.774974 0.000
+ underprediction dispersion log_score mad ae_median se_mean
+ <num> <num> <num> <num> <num> <num>
+ 1: 0.154 0.469424 1.887790 1.4826 1 0.367236
+ 2: 0.000 0.546522 2.231108 2.9652 3 8.479744
+ 3: 0.000 0.574624 1.862234 2.9652 0 0.142884
+ 4: 0.000 0.708469 2.175798 2.9652 3 9.284209
+ 5: 0.000 0.786579 2.489797 2.9652 3 15.327225
+ ---
+220: 1.400 0.345842 2.694333 1.4826 3 5.597956
+221: 1.684 0.288866 2.871442 1.4826 3 6.687396
+222: 2.832 0.260738 4.339301 1.4826 4 13.883076
+223: 2.062 0.256689 3.239324 1.4826 3 8.265625
+224: 0.496 0.278974 1.814469 1.4826 1 0.948676
+
+
+
+
+
+
+
+
+Learning more about the output of score()
+
+
+
+
+
+
See the documentation for get_metrics.forecast_sample()[https://epiforecasts.io/scoringutils/reference/get_metrics.forecast_sample.html] for information on the default metrics for forecasts that are represented as samples (in our case the samples generated by the stan model).
+
+
+
+
+
At a glance
+
Before we look in detail at the scores, we can use summarise_scores to get a quick overview of the scores. Don’t worry if you don’t understand all the scores yet, we will go some of them in more detail in the next section and you can find more information in the {scoringutils} documentation.
Before we look in detail at the scores, what do you think the scores are telling you?
+
+
+
+
+
Continuous ranked probability score
+
+
What is the Continuous Ranked Probability Score (CRPS)?
+
The Continuous Ranked Probability Score (CRPS) is a proper scoring rule used to evaluate the accuracy of probabilistic forecasts. It is a generalization of the Mean Absolute Error (MAE) to probabilistic forecasts, where the forecast is a distribution rather than a single point estimate (i.e. like ours).
+
The CRPS can be thought about as the combination of two key aspects of forecasting: 1. The accuracy of the forecast in terms of how close the predicted values are to the observed value. 2. The confidence of the forecast in terms of the spread of the predicted values.
+
By balancing these two aspects, the CRPS provides a comprehensive measure of the quality of probabilistic forecasts.
+
+
+
+
+
+
+Key things to note about the CRPS
+
+
+
+
+
Small values are better
+
As it is an absolute scoring rule it can be difficult to use to compare forecasts across scales.
+
+
+
+
+
+
+
+
+
+Mathematical Definition (optional)
+
+
+
+
+
+
For distributions with a finite first moment (a mean exists and it is finite), the CRPS can be expressed as:
where \(X\) and \(X'\) are independent random variables sampled from the distribution \(D\). To calculate this we simply replace \(X\) and \(X'\) by samples from our posterior distribution and sum over all possible combinations.
+
This equation can be broke down into the two components:
+
+
Breakdown of the Components
+
+
Expected Absolute Error Between Forecast and Observation: \(\mathbb{E}_{X \sim D}[|X - y|]\) This term represents the average absolute difference between the values predicted by the forecasted distribution \(D\) and the actual observed value \(y\). It measures how far, on average, the forecasted values are from the observed value. A smaller value indicates that the forecasted distribution is closer to the observed value.
+
Expected Absolute Error Between Two Forecasted Values: \(\frac{1}{2} \mathbb{E}_{X, X' \sim D}[|X - X'|]\) This term represents the average absolute difference between two independent samples from the forecasted distribution \(D\). It measures the internal variability or spread of the forecasted distribution. A larger value indicates a wider spread of the forecasted values.
+
+
+
+
Interpretation
+
+
First Term (\(\mathbb{E}_{X \sim D}[|X - y|]\)): This term penalizes the forecast based on how far the predicted values are from the observed value. It ensures that the forecast is accurate in terms of proximity to the actual observation.
+
Second Term (\(\frac{1}{2} \mathbb{E}_{X, X' \sim D}[|X - X'|]\)): This term accounts for the spread of the forecasted distribution. It penalizes forecasts that are too uncertain or have a wide spread. By subtracting this term, the CRPS rewards forecasts that are not only accurate but also confident (i.e., have a narrow spread).
+
+
+
+
+
+
Whilst the CRPS is a very useful metric it can be difficult to interpret in isolation. It is often useful to compare the CRPS of different models or to compare the CRPS of the same model under different conditions. For example, lets compare the CRPS across different forecast horizons.
+
+
sc_scores |>
+summarise_scores(by ="horizon") |>
+ggplot(aes(x = horizon, y = crps)) +
+geom_point() +
+labs(title ="CRPS by daily forecast horizon",
+subtitle ="Summarised across all forecasts")
+
+
+
+
+
+
+
+
+
and at different time points.
+
+
sc_scores |>
+summarise_scores(by ="target_day") |>
+ggplot(aes(x = target_day, y = crps)) +
+geom_point() +
+labs(title ="CRPS by forecast start date",
+subtitle ="Summarised across all forecasts", x ="forecast date")
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Take 5 minutes
+
+
+
+
How do the CRPS scores change based on forecast date? How do the CRPS scores change with forecast horizon? What does this tell you about the model?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
+
The CRPS scores increase for forecast dates where incidence is higher.
+
The CRPS scores increase with forecast horizon.
+
As the CRPS is an absolute measure it is hard to immediately know if the CRPS increasing with forecast date indicates that the model is performing worse.
+
However, the CRPS increasing with forecast horizon is a sign that the model is struggling to capture the longer term dynamics of the epidemic.
+
+
+
+
+
+
+
+
PIT histograms
+
As well as the CRPS we can also look at the calibration and bias of the model. Calibration is the agreement between the forecast probabilities and the observed frequencies. Bias is a measure of how likely the model is to over or under predict the observed values.
+
There are many ways to assess calibration and bias but one common way is to use a probability integral transform (PIT) histogram. This is a histogram of the cumulative distribution of function of a forecast evaluated at the observed value.
+
+
+
+
+
+
+Interpreting the PIT histogram
+
+
+
+
+
Ideally PIT histograms should be uniform.
+
If is a U shape then the model is overconfident and if it is an inverted U shape then the model is underconfident.
+
If it is skewed then the model is biased towards the direction of the skew.
+
+
+
+
+
+
+
+
+
+Mathematical Definition (optional)
+
+
+
+
+
+
Continuous Case
+
For a continuous random variable \(X\) with cumulative distribution function (CDF) \(F_X\), the PIT is defined as:
+
\[
+Y = F_X(X)
+\]
+
where \(Y\) is uniformly distributed on \([0, 1]\).
+
+
Integer Case
+
When dealing with integer forecasts, the standard PIT does not yield a uniform distribution even if the forecasts are perfectly calibrated. To address this, a randomized version of the PIT can be used (Czado, Gneiting, and Held 2009). For an integer-valued random variable \(X\) with CDF \(F_X\), the randomized PIT is defined as:
+
\[
+U = F_X(k) + v \cdot (F_X(k) - F_X(k-1))
+\]
+
where:
+
+
\(k\) is the observed integer value.
+
\(F_X(k)\) is the CDF evaluated at \(k\).
+
\(v\) is a random variable uniformly distributed on \([0, 1]\).
+
+
This transformation ensures that \(U\) is uniformly distributed on \([0, 1]\) if the forecasted distribution \(F_X\) is correctly specified.
+
+
+
+
+
Let’s first look at the overall PIT histogram.
+
+
sc_forecasts |>
+get_pit_histogram() |>
+ggplot(aes(x = mid, y = density)) +
+geom_col() +
+labs(title ="PIT histogram", x ="Quantile", y ="Density")
+
+
+
+
+
+
+
+
+
As before lets look at the PIT histogram by forecast horizon. To save space we will group horizons into a few days each:
What do you think of the PIT histograms? Do they look well calibrated? Do they look biased?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
+
It looks like the model is biased towards overpredicting and that this bias gets worse at longer forecast horizons.
+
Looking over forecast dates it looks like much of this bias is coming from near the outbreak peak where the model is consistently overpredicting but the model is also over predicting at other times.
+
+
+
+
+
+
+
+
Scoring on the log scale
+
We can also score on the logarithmic scale. This can be useful if we are interested in the relative performance of the model at different scales of the data, for example if we are interested in the model’s performance at capturing the exponential growth phase of the epidemic. In some sense scoring in this way can be an approximation of scoring the effective reproduction number estimates. Doing this directly can be difficult as the effective reproduction number is a latent variable and so we cannot directly score it.
+
We again use scoringutils but first transform both the forecasts and observations to the log scale.
Before we look in detail at the scores, what do you think the scores are telling you? How do you think they will differ from the scores on the natural scale?
+
+
+
+
+
CRPS
+
+
log_scores |>
+summarise_scores(by ="horizon") |>
+ggplot(aes(x = horizon, y = crps)) +
+geom_point() +
+labs(title ="CRPS by daily forecast horizon, scored on the log scale")
+
+
+
+
+
+
+
+
+
and across different forecast dates
+
+
log_scores |>
+summarise_scores(by ="target_day") |>
+ggplot(aes(x = target_day, y = crps)) +
+geom_point() +
+labs(title ="CRPS by forecast date, scored on the log scale")
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Take 5 minutes
+
+
+
+
How do the CRPS scores change based on forecast date? How do the CRPS scores change with forecast horizon? What does this tell you about the model?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
+
As for the natural scale CRPS scores increase with forecast horizon but now the increase appears to be linear vs exponential.
+
There has been a reduction in the CRPS scores for forecast dates near the outbreak peak compared to other forecast dates but this is still the period where the model is performing worst.
+
+
+
+
+
+
+
PIT histograms
+
Let’s first look at the overall PIT histogram.
+
+
log_sc_forecasts |>
+get_pit_histogram(by ="model") |>
+ggplot(aes(x = mid, y = density)) +
+geom_col() +
+labs(title ="PIT histogram, scored on the log scale")
+
+
+
+
+
+
+
+
+
As before lets look at the PIT histogram by forecast horizon
+
+
log_sc_forecasts |>
+mutate(group_horizon =case_when(
+ horizon <=3~"1-3",
+ horizon <=7~"4-7",
+ horizon <=14~"8-14"
+ )) |>
+get_pit_histogram(by ="group_horizon") |>
+ggplot(aes(x = mid, y = density)) +
+geom_col() +
+facet_wrap(~group_horizon) +
+labs(title ="PIT by forecast horizon, scored on the log scale")
+
+
+
+
+
+
+
+
+
and then for different forecast dates.
+
+
log_sc_forecasts |>
+get_pit_histogram(by ="target_day") |>
+ggplot(aes(x = mid, y = density)) +
+geom_col() +
+facet_wrap(~target_day) +
+labs(title ="PIT by forecast date, scored on the log scale")
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Take 5 minutes
+
+
+
+
What do you think of the PIT histograms? Do they look well calibrated? Do they look biased?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
+
The overall PIT histograms suggest that the model is less biased to over predict when scored on the log scale than the natural scale, but it is still biased. This makes sense when we think back to the comparison of reproduction number estimates and forecasts we made earlier where the model was consistently over predicting on the reproduction number.
+
By forecast horizon the model is still biased towards over predicting but this bias is less pronounced than on the natural scale.
+
Towards the end and beginning of the forecast period the model appears to be well calibrated on the log scale but is biased towards over predicting in the middle of the forecast period.
+
This matches with our knowledge of the underlying reproduction number which were initially constant and then began to decrease only to stabilise towards the end of the outbreak.
+
+
+
+
+
+
+
+
+
Going further
+
+
In which other ways could we summarise the performance of the forecasts?
+
What other metrics could we use?
+
There is no one-size-fits-all approach to forecast evaluation, often you will need to use a combination of metrics to understand the performance of your model and typically the metrics you use will depend on the context of the forecast. What attributes of the forecast are most important to you?
+
There are many other metrics that can be used to evaluate forecasts. The documentation for the {scoringutils} package has a good overview of these metrics and how to use them.
+
One useful way to think about evaluating forecasts is to consider exploring the scores as a data analysis in its own right. For example, you could look at how the scores change over time, how they change with different forecast horizons, or how they change with different models. This can be a useful way to understand the strengths and weaknesses of your model. Explore some of these aspects using the scores from this session.
+
+
+
+
Wrap up
+
+
+
References
+
+
+Bosse, Nikos I., Sam Abbott, Anne Cori, Edwin van Leeuwen, Johannes Bracher, and Sebastian Funk. 2023. “Scoring Epidemiological Forecasts on Transformed Scales.” Edited by James M McCaw. PLOS Computational Biology 19 (8): e1011393. https://doi.org/10.1371/journal.pcbi.1011393.
+
+Gneiting, Tilmann, and Adrian E Raftery. 2007. “Strictly Proper Scoring Rules, Prediction, and Estimation.”Journal of the American Statistical Association 102 (477): 359–78. https://doi.org/10.1198/016214506000001437.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/sessions/forecast-evaluation_files/figure-html/log-pit-histogram-1.png b/sessions/forecast-evaluation_files/figure-html/log-pit-histogram-1.png
new file mode 100644
index 0000000..4f74765
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/log-pit-histogram-1.png differ
diff --git a/sessions/forecast-evaluation_files/figure-html/log-pit-histogram-date-1.png b/sessions/forecast-evaluation_files/figure-html/log-pit-histogram-date-1.png
new file mode 100644
index 0000000..6776348
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/log-pit-histogram-date-1.png differ
diff --git a/sessions/forecast-evaluation_files/figure-html/log-pit-histogram-horizon-1.png b/sessions/forecast-evaluation_files/figure-html/log-pit-histogram-horizon-1.png
new file mode 100644
index 0000000..b6da822
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/log-pit-histogram-horizon-1.png differ
diff --git a/sessions/forecast-evaluation_files/figure-html/pit-histogram-1.png b/sessions/forecast-evaluation_files/figure-html/pit-histogram-1.png
new file mode 100644
index 0000000..a0ef623
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/pit-histogram-1.png differ
diff --git a/sessions/forecast-evaluation_files/figure-html/pit-histogram-date-1.png b/sessions/forecast-evaluation_files/figure-html/pit-histogram-date-1.png
new file mode 100644
index 0000000..55001e1
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/pit-histogram-date-1.png differ
diff --git a/sessions/forecast-evaluation_files/figure-html/pit-histogram-horizon-1.png b/sessions/forecast-evaluation_files/figure-html/pit-histogram-horizon-1.png
new file mode 100644
index 0000000..050b5c5
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/pit-histogram-horizon-1.png differ
diff --git a/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-4-1.png b/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-4-1.png
new file mode 100644
index 0000000..2e8df5a
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-4-1.png differ
diff --git a/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-5-1.png b/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-5-1.png
new file mode 100644
index 0000000..8c7a26c
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-5-1.png differ
diff --git a/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-7-1.png b/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-7-1.png
new file mode 100644
index 0000000..9da8ff4
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-7-1.png differ
diff --git a/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-8-1.png b/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-8-1.png
new file mode 100644
index 0000000..40054af
Binary files /dev/null and b/sessions/forecast-evaluation_files/figure-html/unnamed-chunk-8-1.png differ
diff --git a/sessions/forecast-visualisation.html b/sessions/forecast-visualisation.html
new file mode 100644
index 0000000..3e53282
--- /dev/null
+++ b/sessions/forecast-visualisation.html
@@ -0,0 +1,1421 @@
+
+
+
+
+
+
+
+
+
+Visualising infectious disease forecasts – UEIFID
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Visualising infectious disease forecasts
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Introduction
+
Epidemiological forecasts are probabilistic statements about what might happen to population disease burden in the future. In this session we will make some simple forecasts using a commonly used infectious disease model, based on the renewal equation. We will see how we can visualise such forecasts, and visually compare them to what really happened.
The aim of this session is to introduce the concept of forecasting and forecast visualisation using a simple model.
+
+
+
+
+
+
+Setup
+
+
+
+
+
+
+
Source file
+
The source file of this session is located at sessions/forecasting-visualisation.qmd.
+
+
+
Libraries used
+
In this session we will use the nfidd package to load a data set of infection times and corresponding forecasts, the dplyr package for data wrangling, and the ggplot2 library for plotting.
The best way to interact with the material is via the Visual Editor of RStudio.
+
+
+
+
+
Initialisation
+
We set a random seed for reproducibility. Setting this ensures that you should get exactly the same results on your computer as we do.
+
+
set.seed(123)
+
+
+
+
+
+
+
+
+
What is forecasting?
+
Forecasting is the process of making predictions about the future based on past and present data. In the context of infectious disease epidemiology, forecasting is usually the process of predicting the future course of some metric of infectious disease incidence or prevalence based on past and present data. Here we focus on forecasting observed data (the number of individuals with new symptom onset) but forecasts can also be made for other quantities of interest such as the number of infections, the reproduction number, or the number of deaths. Epidemiological forecasting is closely related to nowcasting and, when using mechanistic approaches, estimation of the reproduction number. In fact, the model we will use for forecasting could also be used for real-time estimation of the reproduction number and nowcasting, two tasks we won’t have time to go into here but which are discussed in the longer version of this course. The only difference to these tasks is that here we extend the model into the future.
+
+
+
Extending a model into the future
+
We will use a data set of infections we generate using functions from the nfidd R package, and look at forecasts made using a renewal equation model. This model assumes that number infectious people determines the number of future infectious people via the reproduction number \(R_t\).
+
+
+
The reproduction number \(R_t\)
+
The reproduction number \(R_t\) (sometimes called the effective reproduction number) more generally describes the average number of secondary infections caused by a single infectious individual and can vary in time and space as a function of differences in population level susceptibility, changes in behaviour, policy, seasonality etc. The reproduction number \(R_t\) is therefore a more general concept than the basic reproduction number \(R_0\) which represents the average number of secondary infections caused by a single infectious individual in a completely susceptible population.
+
+
+
+
+
+
+
+The discrete renewal equation (optional)
+
+
+
+
+
+
The renewal equation represents a general model of infectious disease transmission which includes the SIR model as a special case. In its basic form it makes no assumption about the specific processes that cause \(R_t\) to have a certain value and/or change over time, but instead it only relates the number of infected people in the population, the current value of the reproduction number and a delay distribution that represents the timings of when individuals infect others relative to when they themselves became infected, the so-called generation time. Mathematically, it can be written as
Here, \(I_t\) is the number of infected individuals on day \(t\), \(R_t\) is the current value of the reproduction number and \(g_i\) is the probability of a secondary infection occurring \(i\) days after the infector became infected themselves, with a maximum \(g_\mathrm{max}\).
+
The forecasting model we visualise in this session is based on \(R_t\) doing a geometric random walk in time. The step size of this geometric random walk is estimated from data, and this is then used to project into the future. For more information on this approach to forecasting see, for example, Funk et al. (2018).
+
+
+
+
+
+
+The geometric random walk model (optional)
+
+
+
+
+
+
We might expect the reproduction number to change smoothly over time (except in situations of drastic change such as a very effective intervention) and to be similar at adjacent time points. We can model this by assuming that the reproduction number at time \(t\) is a random draw from a normal distribution with mean equal to the reproduction number at time \(t-1\) and some standard deviation \(\sigma\). This can be described as a random walk model for the reproduction number. In fact, rather than using this model directly, a better choice might be to use a model where the logarithm of the reproduction number does a random walk, as this will ensure that the reproduction number is always positive and that changes are multiplicative rather than additive (i.e as otherwise the same absolute change in the reproduction number would have a larger effect when the reproduction number is small which likely doesn’t match your intuition for how outbreaks evolve over time). We can write this model as
Here we have placed a prior on the standard deviation of the random walk, which we have assumed to be half-normal (i.e., normal but restricted to being non-negative) with a mean of 0 and a standard deviation of 0.05. This is a so-called weakly informative prior that allows for some variation in the reproduction number over time but not an unrealistic amount. We have also placed a prior on the initial reproduction number, which we have assumed to be log-normally distributed with a mean of -0.1 and a standard deviation of 0.5. This is a weakly informative prior that allows for a wide range of initial reproduction numbers but has a mean of approximately 1. The last line is the geometric random walk.
+
+
Simulating a geometric random walk
+
You can have a look at an R function for performing the geometric random walk:
+
+
geometric_random_walk
+
+
function (init, noise, std)
+{
+ n <- length(noise) + 1
+ x <- numeric(n)
+ x[1] <- init
+ for (i in 2:n) {
+ x[i] <- x[i - 1] + noise[i - 1] * std
+ }
+ return(exp(x))
+}
+<bytecode: 0x55a374bf3f40>
+<environment: namespace:nfidd>
+
+
+
We can use this function to simulate a random walk:
+
+
R <-geometric_random_walk(init =1, noise =rnorm(100), std =0.1)
+data <-tibble(t =seq_along(R), R =exp(R))
+
+ggplot(data, aes(x = t, y = R)) +
+geom_line() +
+labs(title ="Simulated data from a random walk model",
+x ="Time",
+y ="R")
+
+
+
+
+
+
+
+
+
You can repeat this multiple times either with the same parameters or changing some to get a feeling for what this does.
+
+
+
+
+
+
+
+
We will now look at some data from an infectious disease outbreak and forecasts from a renewal equation model. First, we simulate some data using functions from the nfidd package.
This uses a data set of infection times that is included in the R package, and a function to turn these into daily number of observed symptom onsets. Do not worry too much about the details of this. The important thing is that we now have a data set onset_df of symptom onset.
+
+
+
+
+
+
+How does this work?
+
+
+
+
+
+
If you want to see how this works look at the function code by typing simulate_onsets in the console.
+
+
+
+
+
Visualising the forecast
+
We can now visualise a forecast made from the renewal equation model we used for forecasting. Once again, this forecast is provided in the nfidd package which we loaded earlier. We will first extract the forecast and then plot the forecasted number of symptom onsets alongside the observed number of symptom onsets before the forecast was made.
Some important things to note about these forecasts:
+
+
We used a 14 day forecast horizon.
+
Each forecast used all the data up to the forecast date.
+
We generated 1000 predictive posterior samples for each forecast.
+
We started forecasting 3 weeks into the outbreak and then forecast every 7 days until the end of the data (excluding the last 14 days to allow a full forecast).
+
We use the same simulated outbreak data as before:
In this plot, the dots show the data and the lines are forecast trajectories that the model deems plausible and consistent with the data so far.
+
+
+
+
+
+
+Take 5 minutes
+
+
+
+
What do you think of this forecast? Does it match your intuition of how this outbreak will continue? Is there another way you could visualise the forecast that might be more informative?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
+
The forecast mainly predicts things to continue growing as they have been. However, some of the trajectories are going down, indicating that with some probabilities the outbreak might end.
+
Based purely on the data and without knowing anything else about the disease and setting, it is hard to come up with an alternative. The model seems sensible given the available data. In particular, uncertainty increases with increasing distance to the data, which is a sign of a good forecasting model.
+
Instead of visualising plausible trajectories we could visualise a cone with a central forecast and uncertainty around it. We will look at this in the next session as an alternative.
+
+
+
+
+
If we want to know how well a model is doing at forecasting, we can later compare it do the data of what really happened. If we do this, we get the following plot.
What do you think now of this forecast? Did the model do a good job? Again, is there another way you could visualise the forecast that might be more informative?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
+
On the face of it the forecast looks very poor with some very high predictions compared to the data.
+
Based on this visualisation it is hard to tell if the model is doing a good job but it seems like it is not.
+
As outbreaks are generally considered to be exponential processes it might be more informative to plot the forecast on the log scale.
Warning in scale_y_log10(): log-10 transformation introduced infinite values.
+
+
+
+
+
+
+
+
+
+
This should be a lot more informative. We see that for longer forecast horizons the model is not doing a great job of capturing the reduction in symptom onsets. However, we can now see that the model seems to be producing very reasonable forecasts for the first week or so of the forecast. This is a common pattern in forecasting where a model is good at capturing the short term dynamics but struggles with the longer term dynamics.
+
+
+
+
We managed to learn quite a lot about our model’s forecasting limitations just looking at a single forecast using visualisations. However, what if we wanted to quantify how well the model is doing? In order to do that we need to look at multiple forecasts from the model.
+
+
+
Visualising multiple forecasts from a single model
+
As for a single forecast, our first step is to visualise the forecasts as this can give us a good idea of how well the model is doing without having to calculate any metrics.
+
+
rw_forecasts |>
+filter(.draw %in%sample(.draw, 100)) |>
+ggplot(aes(x = day)) +
+geom_line(
+aes(y = .value, group =interaction(.draw, target_day), col = target_day),
+alpha =0.1
+ ) +
+geom_point(
+data = onset_df |>
+filter(day >=21),
+aes(x = day, y = onsets), color ="black") +
+scale_color_binned(type ="viridis") +
+labs(title ="Weekly forecasts of symptom onsets over an outbreak",
+col ="Forecast start day")
+
+
+
+
+
+
+
+
+
As for the single forecast it may be helpful to also plot the forecast on the log scale.
What do you think of these forecasts? Are they any good? How well do they capture changes in trend? Does the uncertainty seem reasonable? Do they seem to under or over predict consistently? Would you visualise the forecast in a different way?
+
+
+
+
+
+
+
+
+Solution
+
+
+
+
+
+
What do you think of these forecasts?
+
+
We think these forecasts are a reasonable place to start but there is definitely room for improvement.
+
+
Are they any good?
+
+
They seem to do a reasonable job of capturing the short term dynamics but struggle with the longer term dynamics.
+
+
How well do they capture changes in trend?
+
+
There is little evidence of the model capturing the reduction in onsets before it begins to show in the data.
+
+
Does the uncertainty seem reasonable?
+
+
On the natural scale it looks like the model often over predicts. Things seem more balanced on the log scale but the model still seems to be overly uncertain.
+
+
Do they seem to under or over predict consistently?
+
+
It looks like the model is consistently over predicting on the natural scale but this is less clear on the log scale.
+
+
+
+
+
+
+
Going further
+
+
What other ways could we visualise the forecasts especially if we had forecasts from multiple models and locations or both?
+
Based on these visualisations how could we improve the model?
+
+
+
+
Wrap up
+
+
+
References
+
+
+Funk, Sebastian, Anton Camacho, Adam J. Kucharski, Rosalind M. Eggo, and W. John Edmunds. 2018. “Real-Time Forecasting of Infectious Disease Dynamics with a Stochastic Semi-Mechanistic Model.”Epidemics 22 (March): 56–61. https://doi.org/10.1016/j.epidem.2016.11.003.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/sessions/forecast-visualisation_files/figure-html/plot-all-forecasts-1.png b/sessions/forecast-visualisation_files/figure-html/plot-all-forecasts-1.png
new file mode 100644
index 0000000..57d482d
Binary files /dev/null and b/sessions/forecast-visualisation_files/figure-html/plot-all-forecasts-1.png differ
diff --git a/sessions/forecast-visualisation_files/figure-html/plot-all-forecasts-log-1.png b/sessions/forecast-visualisation_files/figure-html/plot-all-forecasts-log-1.png
new file mode 100644
index 0000000..e7b6777
Binary files /dev/null and b/sessions/forecast-visualisation_files/figure-html/plot-all-forecasts-log-1.png differ
diff --git a/sessions/forecast-visualisation_files/figure-html/plot_forecast-1.png b/sessions/forecast-visualisation_files/figure-html/plot_forecast-1.png
new file mode 100644
index 0000000..cfe1a4d
Binary files /dev/null and b/sessions/forecast-visualisation_files/figure-html/plot_forecast-1.png differ
diff --git a/sessions/forecast-visualisation_files/figure-html/plot_forecast_log-1.png b/sessions/forecast-visualisation_files/figure-html/plot_forecast_log-1.png
new file mode 100644
index 0000000..2406419
Binary files /dev/null and b/sessions/forecast-visualisation_files/figure-html/plot_forecast_log-1.png differ
diff --git a/sessions/forecast-visualisation_files/figure-html/plot_future_forecast-1.png b/sessions/forecast-visualisation_files/figure-html/plot_future_forecast-1.png
new file mode 100644
index 0000000..2914fed
Binary files /dev/null and b/sessions/forecast-visualisation_files/figure-html/plot_future_forecast-1.png differ
diff --git a/sessions/forecast-visualisation_files/figure-html/simulate-geometric-walk-1.png b/sessions/forecast-visualisation_files/figure-html/simulate-geometric-walk-1.png
new file mode 100644
index 0000000..613a44c
Binary files /dev/null and b/sessions/forecast-visualisation_files/figure-html/simulate-geometric-walk-1.png differ
diff --git a/sessions/introduction-and-course-overview.html b/sessions/introduction-and-course-overview.html
new file mode 100644
index 0000000..4984b81
--- /dev/null
+++ b/sessions/introduction-and-course-overview.html
@@ -0,0 +1,864 @@
+
+
+
+
+
+
+
+
+
+Introduction and course overview – UEIFID
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Visualize and interpret infectious disease forecasts
+
Evaluate forecast performance and limitations
+
Combine multiple forecasts into ensembles
+
+
Predictive modeling has become crucial for public health decision-making and pandemic preparedness. Through this workshop, we aim to make forecast interpretation and evaluation more accessible to academics and public health institutions.
+
+
+
Approach
+
Each session in the course:
+
+
builds on the previous one so that participants will have an overview of forecast evaluation by the end of the course;
+
starts with a short introductory talk;
+
mainly consists of interactive content that participants will work through;
+
has optional/additional material that can be skipped or completed after the course ends;
+
+
For those attending the in-person version the course also:
+
+
has multiple instructors ready to answer questions about this content; if several people have a similar question we may pause the session and discuss it with the group;
+
ends with a wrap-up and discussion where we review the sessions material.
+
+
+
+
Timeline for the course
+
The course consists of four main sessions spread across an afternoon. If you are attending the in-person version of the course, the schedule is available here. If you are studying this material on your own using the website, you can go through it at your own pace.
\n\t\t\t`,K.overlay.querySelector(".close").addEventListener("click",(e=>{ze(),e.preventDefault()}),!1)}}function ze(){return!!K.overlay&&(K.overlay.parentNode.removeChild(K.overlay),K.overlay=null,!0)}function He(){if(K.wrapper&&!oe.isPrintingPDF()){if(!w.disableLayout){g&&!w.embedded&&document.documentElement.style.setProperty("--vh",.01*window.innerHeight+"px");const e=Be(),t=U;qe(w.width,w.height),K.slides.style.width=e.width+"px",K.slides.style.height=e.height+"px",U=Math.min(e.presentationWidth/e.width,e.presentationHeight/e.height),U=Math.max(U,w.minScale),U=Math.min(U,w.maxScale),1===U?(K.slides.style.zoom="",K.slides.style.left="",K.slides.style.top="",K.slides.style.bottom="",K.slides.style.right="",xe({layout:""})):(K.slides.style.zoom="",K.slides.style.left="50%",K.slides.style.top="50%",K.slides.style.bottom="auto",K.slides.style.right="auto",xe({layout:"translate(-50%, -50%) scale("+U+")"}));const i=Array.from(K.wrapper.querySelectorAll(".slides section"));for(let t=0,s=i.length;t .stretch, section > .r-stretch").forEach((t=>{let s=c(t,i);if(/(img|video)/gi.test(t.nodeName)){const i=t.naturalWidth||t.videoWidth,a=t.naturalHeight||t.videoHeight,n=Math.min(e/i,s/a);t.style.width=i*n+"px",t.style.height=a*n+"px"}else t.style.width=e+"px",t.style.height=s+"px"}))}function Be(e,t){const i={width:w.width,height:w.height,presentationWidth:e||K.wrapper.offsetWidth,presentationHeight:t||K.wrapper.offsetHeight};return i.presentationWidth-=i.presentationWidth*w.margin,i.presentationHeight-=i.presentationHeight*w.margin,"string"==typeof i.width&&/%$/.test(i.width)&&(i.width=parseInt(i.width,10)/100*i.presentationWidth),"string"==typeof i.height&&/%$/.test(i.height)&&(i.height=parseInt(i.height,10)/100*i.presentationHeight),i}function Oe(e,t){"object"==typeof e&&"function"==typeof e.setAttribute&&e.setAttribute("data-previous-indexv",t||0)}function Ue(e){if("object"==typeof e&&"function"==typeof e.setAttribute&&e.classList.contains("stack")){const t=e.hasAttribute("data-start-indexv")?"data-start-indexv":"data-previous-indexv";return parseInt(e.getAttribute(t)||0,10)}return 0}function We(e=m){return e&&e.parentNode&&!!e.parentNode.nodeName.match(/section/i)}function Ke(){return!(!m||!We(m))&&!m.nextElementSibling}function Ve(){return 0===u&&0===v}function $e(){return!!m&&(!m.nextElementSibling&&(!We(m)||!m.parentNode.nextElementSibling))}function je(){if(w.pause){const e=K.wrapper.classList.contains("paused");Rt(),K.wrapper.classList.add("paused"),!1===e&&Pe({type:"paused"})}}function Xe(){const e=K.wrapper.classList.contains("paused");K.wrapper.classList.remove("paused"),Et(),e&&Pe({type:"resumed"})}function Ye(e){"boolean"==typeof e?e?je():Xe():_e()?Xe():je()}function _e(){return K.wrapper.classList.contains("paused")}function Je(e){"boolean"==typeof e?e?At():St():Y?At():St()}function Qe(){return!(!$||Y)}function Ze(e,t,i,s){if(Pe({type:"beforeslidechange",data:{indexh:void 0===e?u:e,indexv:void 0===t?v:t,origin:s}}).defaultPrevented)return;p=m;const a=K.wrapper.querySelectorAll(".slides>section");if(0===a.length)return;void 0!==t||ee.isActive()||(t=Ue(a[e])),p&&p.parentNode&&p.parentNode.classList.contains("stack")&&Oe(p.parentNode,v);const n=I.concat();I.length=0;let r=u||0,o=v||0;u=st(".slides>section",void 0===e?u:e),v=st(".slides>section.present>section",void 0===t?v:t);let l=u!==r||v!==o;l||(p=null);let d=a[u],c=d.querySelectorAll("section");m=c[v]||d;let h=!1;l&&p&&m&&!ee.isActive()&&(p.hasAttribute("data-auto-animate")&&m.hasAttribute("data-auto-animate")&&p.getAttribute("data-auto-animate-id")===m.getAttribute("data-auto-animate-id")&&!(u>r||v>o?m:p).hasAttribute("data-auto-animate-restart")&&(h=!0,K.slides.classList.add("disable-slide-transitions")),V="running"),at(),He(),ee.isActive()&&ee.update(),void 0!==i&&G.goto(i),p&&p!==m&&(p.classList.remove("present"),p.setAttribute("aria-hidden","true"),Ve()&&setTimeout((()=>{ut().forEach((e=>{Oe(e,0)}))}),0));e:for(let e=0,t=I.length;e{fe(be(m))})),ae.update(),se.update(),ce.update(),Z.update(),Z.updateParallax(),J.update(),G.update(),ie.writeURL(),Et(),h&&(setTimeout((()=>{K.slides.classList.remove("disable-slide-transitions")}),0),w.autoAnimate&&Q.run(p,m))}function Ge(){Ae(),Se(),He(),$=w.autoSlide,Et(),Z.create(),ie.writeURL(),G.sortAll(),se.update(),ae.update(),at(),ce.update(),ce.updateVisibility(),Z.update(!0),J.update(),_.formatEmbeddedContent(),!1===w.autoPlayMedia?_.stopEmbeddedContent(m,{unloadIframes:!1}):_.startEmbeddedContent(m),ee.isActive()&&ee.layout()}function et(e=m){Z.sync(e),G.sync(e),_.load(e),Z.update(),ce.update()}function tt(){ct().forEach((e=>{t(e,"section").forEach(((e,t)=>{t>0&&(e.classList.remove("present"),e.classList.remove("past"),e.classList.add("future"),e.setAttribute("aria-hidden","true"))}))}))}function it(e=ct()){e.forEach(((t,i)=>{let s=e[Math.floor(Math.random()*e.length)];s.parentNode===t.parentNode&&t.parentNode.insertBefore(t,s);let a=t.querySelectorAll("section");a.length&&it(a)}))}function st(e,i){let s=t(K.wrapper,e),a=s.length,n=oe.isPrintingPDF();if(a){w.loop&&(i%=a)<0&&(i=a+i),i=Math.max(Math.min(i,a-1),0);for(let e=0;e{e.classList.add("visible"),e.classList.remove("current-fragment")}))):e>i&&(a.classList.add(r?"past":"future"),w.fragments&&t(a,".fragment.visible").forEach((e=>{e.classList.remove("visible","current-fragment")})))}let e=s[i],r=e.classList.contains("present");e.classList.add("present"),e.removeAttribute("hidden"),e.removeAttribute("aria-hidden"),r||Pe({target:e,type:"visible",bubbles:!1});let o=e.getAttribute("data-state");o&&(I=I.concat(o.split(" ")))}else i=0;return i}function at(){let e,i,s=ct(),a=s.length;if(a&&void 0!==u){let n=ee.isActive()?10:w.viewDistance;g&&(n=ee.isActive()?6:w.mobileViewDistance),oe.isPrintingPDF()&&(n=Number.MAX_VALUE);for(let r=0;rsection"),i=K.wrapper.querySelectorAll(".slides>section.present>section"),s={left:u>0,right:u0,down:v1&&(s.left=!0,s.right=!0),i.length>1&&(s.up=!0,s.down=!0)),t.length>1&&"linear"===w.navigationMode&&(s.right=s.right||s.down,s.left=s.left||s.up),!0===e){let e=G.availableRoutes();s.left=s.left||e.prev,s.up=s.up||e.prev,s.down=s.down||e.next,s.right=s.right||e.next}if(w.rtl){let e=s.left;s.left=s.right,s.right=e}return s}function rt(e=m){let t=ct(),i=0;e:for(let s=0;s0){let i=.9;t+=m.querySelectorAll(".fragment.visible").length/e.length*i}}return Math.min(t/(e-1),1)}function lt(e){let i,s=u,a=v;if(e){let i=We(e),n=i?e.parentNode:e,r=ct();s=Math.max(r.indexOf(n),0),a=void 0,i&&(a=Math.max(t(e.parentNode,"section").indexOf(e),0))}if(!e&&m){if(m.querySelectorAll(".fragment").length>0){let e=m.querySelector(".current-fragment");i=e&&e.hasAttribute("data-fragment-index")?parseInt(e.getAttribute("data-fragment-index"),10):m.querySelectorAll(".fragment.visible").length-1}}return{h:s,v:a,f:i}}function dt(){return t(K.wrapper,'.slides section:not(.stack):not([data-visibility="uncounted"])')}function ct(){return t(K.wrapper,".slides>section")}function ht(){return t(K.wrapper,".slides>section>section")}function ut(){return t(K.wrapper,".slides>section.stack")}function gt(){return ct().length>1}function vt(){return ht().length>1}function pt(){return dt().map((e=>{let t={};for(let i=0;i{e.hasAttribute("data-autoplay")&&$&&1e3*e.duration/e.playbackRate>$&&($=1e3*e.duration/e.playbackRate+1e3)}))),!$||Y||_e()||ee.isActive()||$e()&&!G.availableRoutes().next&&!0!==w.loop||(j=setTimeout((()=>{"function"==typeof w.autoSlideMethod?w.autoSlideMethod():Nt(),Et()}),$),X=Date.now()),f&&f.setPlaying(-1!==j)}}function Rt(){clearTimeout(j),j=-1}function St(){$&&!Y&&(Y=!0,Pe({type:"autoslidepaused"}),clearTimeout(j),f&&f.setPlaying(!1))}function At(){$&&Y&&(Y=!1,Pe({type:"autoslideresumed"}),Et())}function kt({skipFragments:e=!1}={}){A.hasNavigatedHorizontally=!0,w.rtl?(ee.isActive()||e||!1===G.next())&&nt().left&&Ze(u+1,"grid"===w.navigationMode?v:void 0):(ee.isActive()||e||!1===G.prev())&&nt().left&&Ze(u-1,"grid"===w.navigationMode?v:void 0)}function Lt({skipFragments:e=!1}={}){A.hasNavigatedHorizontally=!0,w.rtl?(ee.isActive()||e||!1===G.prev())&&nt().right&&Ze(u-1,"grid"===w.navigationMode?v:void 0):(ee.isActive()||e||!1===G.next())&&nt().right&&Ze(u+1,"grid"===w.navigationMode?v:void 0)}function Ct({skipFragments:e=!1}={}){(ee.isActive()||e||!1===G.prev())&&nt().up&&Ze(u,v-1)}function xt({skipFragments:e=!1}={}){A.hasNavigatedVertically=!0,(ee.isActive()||e||!1===G.next())&&nt().down&&Ze(u,v+1)}function Pt({skipFragments:e=!1}={}){if(e||!1===G.prev())if(nt().up)Ct({skipFragments:e});else{let i;if(i=w.rtl?t(K.wrapper,".slides>section.future").pop():t(K.wrapper,".slides>section.past").pop(),i&&i.classList.contains("stack")){let e=i.querySelectorAll("section").length-1||void 0;Ze(u-1,e)}else kt({skipFragments:e})}}function Nt({skipFragments:e=!1}={}){if(A.hasNavigatedHorizontally=!0,A.hasNavigatedVertically=!0,e||!1===G.next()){let t=nt();t.down&&t.right&&w.loop&&Ke()&&(t.down=!1),t.down?xt({skipFragments:e}):w.rtl?kt({skipFragments:e}):Lt({skipFragments:e})}}function Mt(e){w.autoSlideStoppable&&St()}function Dt(e){let t=e.data;if("string"==typeof t&&"{"===t.charAt(0)&&"}"===t.charAt(t.length-1)&&(t=JSON.parse(t),t.method&&"function"==typeof h[t.method]))if(!1===R.test(t.method)){const e=h[t.method].apply(h,t.args);Ne("callback",{method:t.method,result:e})}else console.warn('reveal.js: "'+t.method+'" is is blacklisted from the postMessage API')}function It(e){"running"===V&&/section/gi.test(e.target.nodeName)&&(V="idle",Pe({type:"slidetransitionend",data:{indexh:u,indexv:v,previousSlide:p,currentSlide:m}}))}function Tt(e){const t=r(e.target,'a[href^="#"]');if(t){const i=t.getAttribute("href"),s=ie.getIndicesFromHash(i);s&&(h.slide(s.h,s.v,s.f),e.preventDefault())}}function Ft(e){He()}function zt(e){!1===document.hidden&&document.activeElement!==document.body&&("function"==typeof document.activeElement.blur&&document.activeElement.blur(),document.body.focus())}function Ht(e){(document.fullscreenElement||document.webkitFullscreenElement)===K.wrapper&&(e.stopImmediatePropagation(),setTimeout((()=>{h.layout(),h.focus.focus()}),1))}function qt(e){if(e.currentTarget&&e.currentTarget.hasAttribute("href")){let t=e.currentTarget.getAttribute("href");t&&(Ie(t),e.preventDefault())}}function Bt(e){$e()&&!1===w.loop?(Ze(0,0),At()):Y?At():St()}const Ot={VERSION:"4.3.1",initialize:he,configure:Re,destroy:ke,sync:Ge,syncSlide:et,syncFragments:G.sync.bind(G),slide:Ze,left:kt,right:Lt,up:Ct,down:xt,prev:Pt,next:Nt,navigateLeft:kt,navigateRight:Lt,navigateUp:Ct,navigateDown:xt,navigatePrev:Pt,navigateNext:Nt,navigateFragment:G.goto.bind(G),prevFragment:G.prev.bind(G),nextFragment:G.next.bind(G),on:Le,off:Ce,addEventListener:Le,removeEventListener:Ce,layout:He,shuffle:it,availableRoutes:nt,availableFragments:G.availableRoutes.bind(G),toggleHelp:Te,toggleOverview:ee.toggle.bind(ee),togglePause:Ye,toggleAutoSlide:Je,isFirstSlide:Ve,isLastSlide:$e,isLastVerticalSlide:Ke,isVerticalSlide:We,isPaused:_e,isAutoSliding:Qe,isSpeakerNotes:ce.isSpeakerNotesWindow.bind(ce),isOverview:ee.isActive.bind(ee),isFocused:le.isFocused.bind(le),isPrintingPDF:oe.isPrintingPDF.bind(oe),isReady:()=>S,loadSlide:_.load.bind(_),unloadSlide:_.unload.bind(_),showPreview:Ie,hidePreview:ze,addEventListeners:Se,removeEventListeners:Ae,dispatchEvent:Pe,getState:yt,setState:wt,getProgress:ot,getIndices:lt,getSlidesAttributes:pt,getSlidePastCount:rt,getTotalSlides:mt,getSlide:ft,getPreviousSlide:()=>p,getCurrentSlide:()=>m,getSlideBackground:bt,getSlideNotes:ce.getSlideNotes.bind(ce),getSlides:dt,getHorizontalSlides:ct,getVerticalSlides:ht,hasHorizontalSlides:gt,hasVerticalSlides:vt,hasNavigatedHorizontally:()=>A.hasNavigatedHorizontally,hasNavigatedVertically:()=>A.hasNavigatedVertically,addKeyBinding:te.addKeyBinding.bind(te),removeKeyBinding:te.removeKeyBinding.bind(te),triggerKey:te.triggerKey.bind(te),registerKeyboardShortcut:te.registerKeyboardShortcut.bind(te),getComputedSlideSize:Be,getScale:()=>U,getConfig:()=>w,getQueryHash:d,getSlidePath:ie.getHash.bind(ie),getRevealElement:()=>n,getSlidesElement:()=>K.slides,getViewportElement:()=>K.viewport,getBackgroundsElement:()=>Z.element,registerPlugin:re.registerPlugin.bind(re),hasPlugin:re.hasPlugin.bind(re),getPlugin:re.getPlugin.bind(re),getPlugins:re.getRegisteredPlugins.bind(re)};return e(h,{...Ot,announceStatus:fe,getStatusText:be,print:oe,focus:le,progress:ae,controls:se,location:ie,overview:ee,fragments:G,slideContent:_,slideNumber:J,onUserInput:Mt,closeOverlay:ze,updateSlidesVisibility:at,layoutSlideContents:qe,transformSlides:xe,cueAutoSlide:Et,cancelAutoSlide:Rt}),Ot}let W=U,K=[];W.initialize=e=>(Object.assign(W,new U(document.querySelector(".reveal"),e)),K.map((e=>e(W))),W.initialize()),["configure","on","off","addEventListener","removeEventListener","registerPlugin"].forEach((e=>{W[e]=(...t)=>{K.push((i=>i[e].call(null,...t)))}})),W.isReady=()=>!1,W.VERSION="4.3.1";export default W;
+//# sourceMappingURL=reveal.esm.js.map
diff --git a/site_libs/revealjs/dist/reveal.esm.js.map b/site_libs/revealjs/dist/reveal.esm.js.map
new file mode 100644
index 0000000..286c75a
--- /dev/null
+++ b/site_libs/revealjs/dist/reveal.esm.js.map
@@ -0,0 +1 @@
+{"version":3,"file":"reveal.esm.js","sources":["../js/utils/util.js","../js/utils/device.js","../node_modules/fitty/dist/fitty.module.js","../js/controllers/slidecontent.js","../js/controllers/slidenumber.js","../js/utils/color.js","../js/controllers/backgrounds.js","../js/utils/constants.js","../js/controllers/autoanimate.js","../js/controllers/fragments.js","../js/controllers/overview.js","../js/controllers/keyboard.js","../js/controllers/location.js","../js/controllers/controls.js","../js/controllers/progress.js","../js/controllers/pointer.js","../js/utils/loader.js","../js/controllers/plugins.js","../js/controllers/print.js","../js/controllers/touch.js","../js/controllers/focus.js","../js/controllers/notes.js","../js/components/playback.js","../js/config.js","../js/reveal.js","../js/index.js"],"sourcesContent":["/**\n * Extend object a with the properties of object b.\n * If there's a conflict, object b takes precedence.\n *\n * @param {object} a\n * @param {object} b\n */\nexport const extend = ( a, b ) => {\n\n\tfor( let i in b ) {\n\t\ta[ i ] = b[ i ];\n\t}\n\n\treturn a;\n\n}\n\n/**\n * querySelectorAll but returns an Array.\n */\nexport const queryAll = ( el, selector ) => {\n\n\treturn Array.from( el.querySelectorAll( selector ) );\n\n}\n\n/**\n * classList.toggle() with cross browser support\n */\nexport const toggleClass = ( el, className, value ) => {\n\tif( value ) {\n\t\tel.classList.add( className );\n\t}\n\telse {\n\t\tel.classList.remove( className );\n\t}\n}\n\n/**\n * Utility for deserializing a value.\n *\n * @param {*} value\n * @return {*}\n */\nexport const deserialize = ( value ) => {\n\n\tif( typeof value === 'string' ) {\n\t\tif( value === 'null' ) return null;\n\t\telse if( value === 'true' ) return true;\n\t\telse if( value === 'false' ) return false;\n\t\telse if( value.match( /^-?[\\d\\.]+$/ ) ) return parseFloat( value );\n\t}\n\n\treturn value;\n\n}\n\n/**\n * Measures the distance in pixels between point a\n * and point b.\n *\n * @param {object} a point with x/y properties\n * @param {object} b point with x/y properties\n *\n * @return {number}\n */\nexport const distanceBetween = ( a, b ) => {\n\n\tlet dx = a.x - b.x,\n\t\tdy = a.y - b.y;\n\n\treturn Math.sqrt( dx*dx + dy*dy );\n\n}\n\n/**\n * Applies a CSS transform to the target element.\n *\n * @param {HTMLElement} element\n * @param {string} transform\n */\nexport const transformElement = ( element, transform ) => {\n\n\telement.style.transform = transform;\n\n}\n\n/**\n * Element.matches with IE support.\n *\n * @param {HTMLElement} target The element to match\n * @param {String} selector The CSS selector to match\n * the element against\n *\n * @return {Boolean}\n */\nexport const matches = ( target, selector ) => {\n\n\tlet matchesMethod = target.matches || target.matchesSelector || target.msMatchesSelector;\n\n\treturn !!( matchesMethod && matchesMethod.call( target, selector ) );\n\n}\n\n/**\n * Find the closest parent that matches the given\n * selector.\n *\n * @param {HTMLElement} target The child element\n * @param {String} selector The CSS selector to match\n * the parents against\n *\n * @return {HTMLElement} The matched parent or null\n * if no matching parent was found\n */\nexport const closest = ( target, selector ) => {\n\n\t// Native Element.closest\n\tif( typeof target.closest === 'function' ) {\n\t\treturn target.closest( selector );\n\t}\n\n\t// Polyfill\n\twhile( target ) {\n\t\tif( matches( target, selector ) ) {\n\t\t\treturn target;\n\t\t}\n\n\t\t// Keep searching\n\t\ttarget = target.parentNode;\n\t}\n\n\treturn null;\n\n}\n\n/**\n * Handling the fullscreen functionality via the fullscreen API\n *\n * @see http://fullscreen.spec.whatwg.org/\n * @see https://developer.mozilla.org/en-US/docs/DOM/Using_fullscreen_mode\n */\nexport const enterFullscreen = element => {\n\n\telement = element || document.documentElement;\n\n\t// Check which implementation is available\n\tlet requestMethod = element.requestFullscreen ||\n\t\t\t\t\t\telement.webkitRequestFullscreen ||\n\t\t\t\t\t\telement.webkitRequestFullScreen ||\n\t\t\t\t\t\telement.mozRequestFullScreen ||\n\t\t\t\t\t\telement.msRequestFullscreen;\n\n\tif( requestMethod ) {\n\t\trequestMethod.apply( element );\n\t}\n\n}\n\n/**\n * Creates an HTML element and returns a reference to it.\n * If the element already exists the existing instance will\n * be returned.\n *\n * @param {HTMLElement} container\n * @param {string} tagname\n * @param {string} classname\n * @param {string} innerHTML\n *\n * @return {HTMLElement}\n */\nexport const createSingletonNode = ( container, tagname, classname, innerHTML='' ) => {\n\n\t// Find all nodes matching the description\n\tlet nodes = container.querySelectorAll( '.' + classname );\n\n\t// Check all matches to find one which is a direct child of\n\t// the specified container\n\tfor( let i = 0; i < nodes.length; i++ ) {\n\t\tlet testNode = nodes[i];\n\t\tif( testNode.parentNode === container ) {\n\t\t\treturn testNode;\n\t\t}\n\t}\n\n\t// If no node was found, create it now\n\tlet node = document.createElement( tagname );\n\tnode.className = classname;\n\tnode.innerHTML = innerHTML;\n\tcontainer.appendChild( node );\n\n\treturn node;\n\n}\n\n/**\n * Injects the given CSS styles into the DOM.\n *\n * @param {string} value\n */\nexport const createStyleSheet = ( value ) => {\n\n\tlet tag = document.createElement( 'style' );\n\ttag.type = 'text/css';\n\n\tif( value && value.length > 0 ) {\n\t\tif( tag.styleSheet ) {\n\t\t\ttag.styleSheet.cssText = value;\n\t\t}\n\t\telse {\n\t\t\ttag.appendChild( document.createTextNode( value ) );\n\t\t}\n\t}\n\n\tdocument.head.appendChild( tag );\n\n\treturn tag;\n\n}\n\n/**\n * Returns a key:value hash of all query params.\n */\nexport const getQueryHash = () => {\n\n\tlet query = {};\n\n\tlocation.search.replace( /[A-Z0-9]+?=([\\w\\.%-]*)/gi, a => {\n\t\tquery[ a.split( '=' ).shift() ] = a.split( '=' ).pop();\n\t} );\n\n\t// Basic deserialization\n\tfor( let i in query ) {\n\t\tlet value = query[ i ];\n\n\t\tquery[ i ] = deserialize( unescape( value ) );\n\t}\n\n\t// Do not accept new dependencies via query config to avoid\n\t// the potential of malicious script injection\n\tif( typeof query['dependencies'] !== 'undefined' ) delete query['dependencies'];\n\n\treturn query;\n\n}\n\n/**\n * Returns the remaining height within the parent of the\n * target element.\n *\n * remaining height = [ configured parent height ] - [ current parent height ]\n *\n * @param {HTMLElement} element\n * @param {number} [height]\n */\nexport const getRemainingHeight = ( element, height = 0 ) => {\n\n\tif( element ) {\n\t\tlet newHeight, oldHeight = element.style.height;\n\n\t\t// Change the .stretch element height to 0 in order find the height of all\n\t\t// the other elements\n\t\telement.style.height = '0px';\n\n\t\t// In Overview mode, the parent (.slide) height is set of 700px.\n\t\t// Restore it temporarily to its natural height.\n\t\telement.parentNode.style.height = 'auto';\n\n\t\tnewHeight = height - element.parentNode.offsetHeight;\n\n\t\t// Restore the old height, just in case\n\t\telement.style.height = oldHeight + 'px';\n\n\t\t// Clear the parent (.slide) height. .removeProperty works in IE9+\n\t\telement.parentNode.style.removeProperty('height');\n\n\t\treturn newHeight;\n\t}\n\n\treturn height;\n\n}\n\nconst fileExtensionToMimeMap = {\n\t'mp4': 'video/mp4',\n\t'm4a': 'video/mp4',\n\t'ogv': 'video/ogg',\n\t'mpeg': 'video/mpeg',\n\t'webm': 'video/webm'\n}\n\n/**\n * Guess the MIME type for common file formats.\n */\nexport const getMimeTypeFromFile = ( filename='' ) => {\n\treturn fileExtensionToMimeMap[filename.split('.').pop()]\n}","const UA = navigator.userAgent;\n\nexport const isMobile = /(iphone|ipod|ipad|android)/gi.test( UA ) ||\n\t\t\t\t\t\t( navigator.platform === 'MacIntel' && navigator.maxTouchPoints > 1 ); // iPadOS\n\nexport const isChrome = /chrome/i.test( UA ) && !/edge/i.test( UA );\n\nexport const isAndroid = /android/gi.test( UA );","/*\n * fitty v2.3.3 - Snugly resizes text to fit its parent container\n * Copyright (c) 2020 Rik Schennink (https://pqina.nl/)\n */\n'use strict';\n\nObject.defineProperty(exports, \"__esModule\", {\n value: true\n});\n\nvar _extends = Object.assign || function (target) { for (var i = 1; i < arguments.length; i++) { var source = arguments[i]; for (var key in source) { if (Object.prototype.hasOwnProperty.call(source, key)) { target[key] = source[key]; } } } return target; };\n\nexports.default = function (w) {\n\n // no window, early exit\n if (!w) return;\n\n // node list to array helper method\n var toArray = function toArray(nl) {\n return [].slice.call(nl);\n };\n\n // states\n var DrawState = {\n IDLE: 0,\n DIRTY_CONTENT: 1,\n DIRTY_LAYOUT: 2,\n DIRTY: 3\n };\n\n // all active fitty elements\n var fitties = [];\n\n // group all redraw calls till next frame, we cancel each frame request when a new one comes in. If no support for request animation frame, this is an empty function and supports for fitty stops.\n var redrawFrame = null;\n var requestRedraw = 'requestAnimationFrame' in w ? function () {\n w.cancelAnimationFrame(redrawFrame);\n redrawFrame = w.requestAnimationFrame(function () {\n return redraw(fitties.filter(function (f) {\n return f.dirty && f.active;\n }));\n });\n } : function () {};\n\n // sets all fitties to dirty so they are redrawn on the next redraw loop, then calls redraw\n var redrawAll = function redrawAll(type) {\n return function () {\n fitties.forEach(function (f) {\n return f.dirty = type;\n });\n requestRedraw();\n };\n };\n\n // redraws fitties so they nicely fit their parent container\n var redraw = function redraw(fitties) {\n\n // getting info from the DOM at this point should not trigger a reflow, let's gather as much intel as possible before triggering a reflow\n\n // check if styles of all fitties have been computed\n fitties.filter(function (f) {\n return !f.styleComputed;\n }).forEach(function (f) {\n f.styleComputed = computeStyle(f);\n });\n\n // restyle elements that require pre-styling, this triggers a reflow, please try to prevent by adding CSS rules (see docs)\n fitties.filter(shouldPreStyle).forEach(applyStyle);\n\n // we now determine which fitties should be redrawn\n var fittiesToRedraw = fitties.filter(shouldRedraw);\n\n // we calculate final styles for these fitties\n fittiesToRedraw.forEach(calculateStyles);\n\n // now we apply the calculated styles from our previous loop\n fittiesToRedraw.forEach(function (f) {\n applyStyle(f);\n markAsClean(f);\n });\n\n // now we dispatch events for all restyled fitties\n fittiesToRedraw.forEach(dispatchFitEvent);\n };\n\n var markAsClean = function markAsClean(f) {\n return f.dirty = DrawState.IDLE;\n };\n\n var calculateStyles = function calculateStyles(f) {\n\n // get available width from parent node\n f.availableWidth = f.element.parentNode.clientWidth;\n\n // the space our target element uses\n f.currentWidth = f.element.scrollWidth;\n\n // remember current font size\n f.previousFontSize = f.currentFontSize;\n\n // let's calculate the new font size\n f.currentFontSize = Math.min(Math.max(f.minSize, f.availableWidth / f.currentWidth * f.previousFontSize), f.maxSize);\n\n // if allows wrapping, only wrap when at minimum font size (otherwise would break container)\n f.whiteSpace = f.multiLine && f.currentFontSize === f.minSize ? 'normal' : 'nowrap';\n };\n\n // should always redraw if is not dirty layout, if is dirty layout, only redraw if size has changed\n var shouldRedraw = function shouldRedraw(f) {\n return f.dirty !== DrawState.DIRTY_LAYOUT || f.dirty === DrawState.DIRTY_LAYOUT && f.element.parentNode.clientWidth !== f.availableWidth;\n };\n\n // every fitty element is tested for invalid styles\n var computeStyle = function computeStyle(f) {\n\n // get style properties\n var style = w.getComputedStyle(f.element, null);\n\n // get current font size in pixels (if we already calculated it, use the calculated version)\n f.currentFontSize = parseFloat(style.getPropertyValue('font-size'));\n\n // get display type and wrap mode\n f.display = style.getPropertyValue('display');\n f.whiteSpace = style.getPropertyValue('white-space');\n };\n\n // determines if this fitty requires initial styling, can be prevented by applying correct styles through CSS\n var shouldPreStyle = function shouldPreStyle(f) {\n\n var preStyle = false;\n\n // if we already tested for prestyling we don't have to do it again\n if (f.preStyleTestCompleted) return false;\n\n // should have an inline style, if not, apply\n if (!/inline-/.test(f.display)) {\n preStyle = true;\n f.display = 'inline-block';\n }\n\n // to correctly calculate dimensions the element should have whiteSpace set to nowrap\n if (f.whiteSpace !== 'nowrap') {\n preStyle = true;\n f.whiteSpace = 'nowrap';\n }\n\n // we don't have to do this twice\n f.preStyleTestCompleted = true;\n\n return preStyle;\n };\n\n // apply styles to single fitty\n var applyStyle = function applyStyle(f) {\n f.element.style.whiteSpace = f.whiteSpace;\n f.element.style.display = f.display;\n f.element.style.fontSize = f.currentFontSize + 'px';\n };\n\n // dispatch a fit event on a fitty\n var dispatchFitEvent = function dispatchFitEvent(f) {\n f.element.dispatchEvent(new CustomEvent('fit', {\n detail: {\n oldValue: f.previousFontSize,\n newValue: f.currentFontSize,\n scaleFactor: f.currentFontSize / f.previousFontSize\n }\n }));\n };\n\n // fit method, marks the fitty as dirty and requests a redraw (this will also redraw any other fitty marked as dirty)\n var fit = function fit(f, type) {\n return function () {\n f.dirty = type;\n if (!f.active) return;\n requestRedraw();\n };\n };\n\n var init = function init(f) {\n\n // save some of the original CSS properties before we change them\n f.originalStyle = {\n whiteSpace: f.element.style.whiteSpace,\n display: f.element.style.display,\n fontSize: f.element.style.fontSize\n };\n\n // should we observe DOM mutations\n observeMutations(f);\n\n // this is a new fitty so we need to validate if it's styles are in order\n f.newbie = true;\n\n // because it's a new fitty it should also be dirty, we want it to redraw on the first loop\n f.dirty = true;\n\n // we want to be able to update this fitty\n fitties.push(f);\n };\n\n var destroy = function destroy(f) {\n return function () {\n\n // remove from fitties array\n fitties = fitties.filter(function (_) {\n return _.element !== f.element;\n });\n\n // stop observing DOM\n if (f.observeMutations) f.observer.disconnect();\n\n // reset the CSS properties we changes\n f.element.style.whiteSpace = f.originalStyle.whiteSpace;\n f.element.style.display = f.originalStyle.display;\n f.element.style.fontSize = f.originalStyle.fontSize;\n };\n };\n\n // add a new fitty, does not redraw said fitty\n var subscribe = function subscribe(f) {\n return function () {\n if (f.active) return;\n f.active = true;\n requestRedraw();\n };\n };\n\n // remove an existing fitty\n var unsubscribe = function unsubscribe(f) {\n return function () {\n return f.active = false;\n };\n };\n\n var observeMutations = function observeMutations(f) {\n\n // no observing?\n if (!f.observeMutations) return;\n\n // start observing mutations\n f.observer = new MutationObserver(fit(f, DrawState.DIRTY_CONTENT));\n\n // start observing\n f.observer.observe(f.element, f.observeMutations);\n };\n\n // default mutation observer settings\n var mutationObserverDefaultSetting = {\n subtree: true,\n childList: true,\n characterData: true\n };\n\n // default fitty options\n var defaultOptions = {\n minSize: 16,\n maxSize: 512,\n multiLine: true,\n observeMutations: 'MutationObserver' in w ? mutationObserverDefaultSetting : false\n };\n\n // array of elements in, fitty instances out\n function fittyCreate(elements, options) {\n\n // set options object\n var fittyOptions = _extends({}, defaultOptions, options);\n\n // create fitties\n var publicFitties = elements.map(function (element) {\n\n // create fitty instance\n var f = _extends({}, fittyOptions, {\n\n // internal options for this fitty\n element: element,\n active: true\n });\n\n // initialise this fitty\n init(f);\n\n // expose API\n return {\n element: element,\n fit: fit(f, DrawState.DIRTY),\n unfreeze: subscribe(f),\n freeze: unsubscribe(f),\n unsubscribe: destroy(f)\n };\n });\n\n // call redraw on newly initiated fitties\n requestRedraw();\n\n // expose fitties\n return publicFitties;\n }\n\n // fitty creation function\n function fitty(target) {\n var options = arguments.length > 1 && arguments[1] !== undefined ? arguments[1] : {};\n\n\n // if target is a string\n return typeof target === 'string' ?\n\n // treat it as a querySelector\n fittyCreate(toArray(document.querySelectorAll(target)), options) :\n\n // create single fitty\n fittyCreate([target], options)[0];\n }\n\n // handles viewport changes, redraws all fitties, but only does so after a timeout\n var resizeDebounce = null;\n var onWindowResized = function onWindowResized() {\n w.clearTimeout(resizeDebounce);\n resizeDebounce = w.setTimeout(redrawAll(DrawState.DIRTY_LAYOUT), fitty.observeWindowDelay);\n };\n\n // define observe window property, so when we set it to true or false events are automatically added and removed\n var events = ['resize', 'orientationchange'];\n Object.defineProperty(fitty, 'observeWindow', {\n set: function set(enabled) {\n var method = (enabled ? 'add' : 'remove') + 'EventListener';\n events.forEach(function (e) {\n w[method](e, onWindowResized);\n });\n }\n });\n\n // fitty global properties (by setting observeWindow to true the events above get added)\n fitty.observeWindow = true;\n fitty.observeWindowDelay = 100;\n\n // public fit all method, will force redraw no matter what\n fitty.fitAll = redrawAll(DrawState.DIRTY);\n\n // export our fitty function, we don't want to keep it to our selves\n return fitty;\n}(typeof window === 'undefined' ? null : window);","import { extend, queryAll, closest, getMimeTypeFromFile } from '../utils/util.js'\nimport { isMobile } from '../utils/device.js'\n\nimport fitty from 'fitty';\n\n/**\n * Handles loading, unloading and playback of slide\n * content such as images, videos and iframes.\n */\nexport default class SlideContent {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t\tthis.startEmbeddedIframe = this.startEmbeddedIframe.bind( this );\n\n\t}\n\n\t/**\n\t * Should the given element be preloaded?\n\t * Decides based on local element attributes and global config.\n\t *\n\t * @param {HTMLElement} element\n\t */\n\tshouldPreload( element ) {\n\n\t\t// Prefer an explicit global preload setting\n\t\tlet preload = this.Reveal.getConfig().preloadIframes;\n\n\t\t// If no global setting is available, fall back on the element's\n\t\t// own preload setting\n\t\tif( typeof preload !== 'boolean' ) {\n\t\t\tpreload = element.hasAttribute( 'data-preload' );\n\t\t}\n\n\t\treturn preload;\n\t}\n\n\t/**\n\t * Called when the given slide is within the configured view\n\t * distance. Shows the slide element and loads any content\n\t * that is set to load lazily (data-src).\n\t *\n\t * @param {HTMLElement} slide Slide to show\n\t */\n\tload( slide, options = {} ) {\n\n\t\t// Show the slide element\n\t\tslide.style.display = this.Reveal.getConfig().display;\n\n\t\t// Media elements with data-src attributes\n\t\tqueryAll( slide, 'img[data-src], video[data-src], audio[data-src], iframe[data-src]' ).forEach( element => {\n\t\t\tif( element.tagName !== 'IFRAME' || this.shouldPreload( element ) ) {\n\t\t\t\telement.setAttribute( 'src', element.getAttribute( 'data-src' ) );\n\t\t\t\telement.setAttribute( 'data-lazy-loaded', '' );\n\t\t\t\telement.removeAttribute( 'data-src' );\n\t\t\t}\n\t\t} );\n\n\t\t// Media elements with children\n\t\tqueryAll( slide, 'video, audio' ).forEach( media => {\n\t\t\tlet sources = 0;\n\n\t\t\tqueryAll( media, 'source[data-src]' ).forEach( source => {\n\t\t\t\tsource.setAttribute( 'src', source.getAttribute( 'data-src' ) );\n\t\t\t\tsource.removeAttribute( 'data-src' );\n\t\t\t\tsource.setAttribute( 'data-lazy-loaded', '' );\n\t\t\t\tsources += 1;\n\t\t\t} );\n\n\t\t\t// Enable inline video playback in mobile Safari\n\t\t\tif( isMobile && media.tagName === 'VIDEO' ) {\n\t\t\t\tmedia.setAttribute( 'playsinline', '' );\n\t\t\t}\n\n\t\t\t// If we rewrote sources for this video/audio element, we need\n\t\t\t// to manually tell it to load from its new origin\n\t\t\tif( sources > 0 ) {\n\t\t\t\tmedia.load();\n\t\t\t}\n\t\t} );\n\n\n\t\t// Show the corresponding background element\n\t\tlet background = slide.slideBackgroundElement;\n\t\tif( background ) {\n\t\t\tbackground.style.display = 'block';\n\n\t\t\tlet backgroundContent = slide.slideBackgroundContentElement;\n\t\t\tlet backgroundIframe = slide.getAttribute( 'data-background-iframe' );\n\n\t\t\t// If the background contains media, load it\n\t\t\tif( background.hasAttribute( 'data-loaded' ) === false ) {\n\t\t\t\tbackground.setAttribute( 'data-loaded', 'true' );\n\n\t\t\t\tlet backgroundImage = slide.getAttribute( 'data-background-image' ),\n\t\t\t\t\tbackgroundVideo = slide.getAttribute( 'data-background-video' ),\n\t\t\t\t\tbackgroundVideoLoop = slide.hasAttribute( 'data-background-video-loop' ),\n\t\t\t\t\tbackgroundVideoMuted = slide.hasAttribute( 'data-background-video-muted' );\n\n\t\t\t\t// Images\n\t\t\t\tif( backgroundImage ) {\n\t\t\t\t\t// base64\n\t\t\t\t\tif( /^data:/.test( backgroundImage.trim() ) ) {\n\t\t\t\t\t\tbackgroundContent.style.backgroundImage = `url(${backgroundImage.trim()})`;\n\t\t\t\t\t}\n\t\t\t\t\t// URL(s)\n\t\t\t\t\telse {\n\t\t\t\t\t\tbackgroundContent.style.backgroundImage = backgroundImage.split( ',' ).map( background => {\n\t\t\t\t\t\t\treturn `url(${encodeURI(background.trim())})`;\n\t\t\t\t\t\t}).join( ',' );\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\t// Videos\n\t\t\t\telse if ( backgroundVideo && !this.Reveal.isSpeakerNotes() ) {\n\t\t\t\t\tlet video = document.createElement( 'video' );\n\n\t\t\t\t\tif( backgroundVideoLoop ) {\n\t\t\t\t\t\tvideo.setAttribute( 'loop', '' );\n\t\t\t\t\t}\n\n\t\t\t\t\tif( backgroundVideoMuted ) {\n\t\t\t\t\t\tvideo.muted = true;\n\t\t\t\t\t}\n\n\t\t\t\t\t// Enable inline playback in mobile Safari\n\t\t\t\t\t//\n\t\t\t\t\t// Mute is required for video to play when using\n\t\t\t\t\t// swipe gestures to navigate since they don't\n\t\t\t\t\t// count as direct user actions :'(\n\t\t\t\t\tif( isMobile ) {\n\t\t\t\t\t\tvideo.muted = true;\n\t\t\t\t\t\tvideo.setAttribute( 'playsinline', '' );\n\t\t\t\t\t}\n\n\t\t\t\t\t// Support comma separated lists of video sources\n\t\t\t\t\tbackgroundVideo.split( ',' ).forEach( source => {\n\t\t\t\t\t\tlet type = getMimeTypeFromFile( source );\n\t\t\t\t\t\tif( type ) {\n\t\t\t\t\t\t\tvideo.innerHTML += ``;\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tvideo.innerHTML += ``;\n\t\t\t\t\t\t}\n\t\t\t\t\t} );\n\n\t\t\t\t\tbackgroundContent.appendChild( video );\n\t\t\t\t}\n\t\t\t\t// Iframes\n\t\t\t\telse if( backgroundIframe && options.excludeIframes !== true ) {\n\t\t\t\t\tlet iframe = document.createElement( 'iframe' );\n\t\t\t\t\tiframe.setAttribute( 'allowfullscreen', '' );\n\t\t\t\t\tiframe.setAttribute( 'mozallowfullscreen', '' );\n\t\t\t\t\tiframe.setAttribute( 'webkitallowfullscreen', '' );\n\t\t\t\t\tiframe.setAttribute( 'allow', 'autoplay' );\n\n\t\t\t\t\tiframe.setAttribute( 'data-src', backgroundIframe );\n\n\t\t\t\t\tiframe.style.width = '100%';\n\t\t\t\t\tiframe.style.height = '100%';\n\t\t\t\t\tiframe.style.maxHeight = '100%';\n\t\t\t\t\tiframe.style.maxWidth = '100%';\n\n\t\t\t\t\tbackgroundContent.appendChild( iframe );\n\t\t\t\t}\n\t\t\t}\n\n\t\t\t// Start loading preloadable iframes\n\t\t\tlet backgroundIframeElement = backgroundContent.querySelector( 'iframe[data-src]' );\n\t\t\tif( backgroundIframeElement ) {\n\n\t\t\t\t// Check if this iframe is eligible to be preloaded\n\t\t\t\tif( this.shouldPreload( background ) && !/autoplay=(1|true|yes)/gi.test( backgroundIframe ) ) {\n\t\t\t\t\tif( backgroundIframeElement.getAttribute( 'src' ) !== backgroundIframe ) {\n\t\t\t\t\t\tbackgroundIframeElement.setAttribute( 'src', backgroundIframe );\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t}\n\n\t\t}\n\n\t\tthis.layout( slide );\n\n\t}\n\n\t/**\n\t * Applies JS-dependent layout helpers for the given slide,\n\t * if there are any.\n\t */\n\tlayout( slide ) {\n\n\t\t// Autosize text with the r-fit-text class based on the\n\t\t// size of its container. This needs to happen after the\n\t\t// slide is visible in order to measure the text.\n\t\tArray.from( slide.querySelectorAll( '.r-fit-text' ) ).forEach( element => {\n\t\t\tfitty( element, {\n\t\t\t\tminSize: 24,\n\t\t\t\tmaxSize: this.Reveal.getConfig().height * 0.8,\n\t\t\t\tobserveMutations: false,\n\t\t\t\tobserveWindow: false\n\t\t\t} );\n\t\t} );\n\n\t}\n\n\t/**\n\t * Unloads and hides the given slide. This is called when the\n\t * slide is moved outside of the configured view distance.\n\t *\n\t * @param {HTMLElement} slide\n\t */\n\tunload( slide ) {\n\n\t\t// Hide the slide element\n\t\tslide.style.display = 'none';\n\n\t\t// Hide the corresponding background element\n\t\tlet background = this.Reveal.getSlideBackground( slide );\n\t\tif( background ) {\n\t\t\tbackground.style.display = 'none';\n\n\t\t\t// Unload any background iframes\n\t\t\tqueryAll( background, 'iframe[src]' ).forEach( element => {\n\t\t\t\telement.removeAttribute( 'src' );\n\t\t\t} );\n\t\t}\n\n\t\t// Reset lazy-loaded media elements with src attributes\n\t\tqueryAll( slide, 'video[data-lazy-loaded][src], audio[data-lazy-loaded][src], iframe[data-lazy-loaded][src]' ).forEach( element => {\n\t\t\telement.setAttribute( 'data-src', element.getAttribute( 'src' ) );\n\t\t\telement.removeAttribute( 'src' );\n\t\t} );\n\n\t\t// Reset lazy-loaded media elements with children\n\t\tqueryAll( slide, 'video[data-lazy-loaded] source[src], audio source[src]' ).forEach( source => {\n\t\t\tsource.setAttribute( 'data-src', source.getAttribute( 'src' ) );\n\t\t\tsource.removeAttribute( 'src' );\n\t\t} );\n\n\t}\n\n\t/**\n\t * Enforces origin-specific format rules for embedded media.\n\t */\n\tformatEmbeddedContent() {\n\n\t\tlet _appendParamToIframeSource = ( sourceAttribute, sourceURL, param ) => {\n\t\t\tqueryAll( this.Reveal.getSlidesElement(), 'iframe['+ sourceAttribute +'*=\"'+ sourceURL +'\"]' ).forEach( el => {\n\t\t\t\tlet src = el.getAttribute( sourceAttribute );\n\t\t\t\tif( src && src.indexOf( param ) === -1 ) {\n\t\t\t\t\tel.setAttribute( sourceAttribute, src + ( !/\\?/.test( src ) ? '?' : '&' ) + param );\n\t\t\t\t}\n\t\t\t});\n\t\t};\n\n\t\t// YouTube frames must include \"?enablejsapi=1\"\n\t\t_appendParamToIframeSource( 'src', 'youtube.com/embed/', 'enablejsapi=1' );\n\t\t_appendParamToIframeSource( 'data-src', 'youtube.com/embed/', 'enablejsapi=1' );\n\n\t\t// Vimeo frames must include \"?api=1\"\n\t\t_appendParamToIframeSource( 'src', 'player.vimeo.com/', 'api=1' );\n\t\t_appendParamToIframeSource( 'data-src', 'player.vimeo.com/', 'api=1' );\n\n\t}\n\n\t/**\n\t * Start playback of any embedded content inside of\n\t * the given element.\n\t *\n\t * @param {HTMLElement} element\n\t */\n\tstartEmbeddedContent( element ) {\n\n\t\tif( element && !this.Reveal.isSpeakerNotes() ) {\n\n\t\t\t// Restart GIFs\n\t\t\tqueryAll( element, 'img[src$=\".gif\"]' ).forEach( el => {\n\t\t\t\t// Setting the same unchanged source like this was confirmed\n\t\t\t\t// to work in Chrome, FF & Safari\n\t\t\t\tel.setAttribute( 'src', el.getAttribute( 'src' ) );\n\t\t\t} );\n\n\t\t\t// HTML5 media elements\n\t\t\tqueryAll( element, 'video, audio' ).forEach( el => {\n\t\t\t\tif( closest( el, '.fragment' ) && !closest( el, '.fragment.visible' ) ) {\n\t\t\t\t\treturn;\n\t\t\t\t}\n\n\t\t\t\t// Prefer an explicit global autoplay setting\n\t\t\t\tlet autoplay = this.Reveal.getConfig().autoPlayMedia;\n\n\t\t\t\t// If no global setting is available, fall back on the element's\n\t\t\t\t// own autoplay setting\n\t\t\t\tif( typeof autoplay !== 'boolean' ) {\n\t\t\t\t\tautoplay = el.hasAttribute( 'data-autoplay' ) || !!closest( el, '.slide-background' );\n\t\t\t\t}\n\n\t\t\t\tif( autoplay && typeof el.play === 'function' ) {\n\n\t\t\t\t\t// If the media is ready, start playback\n\t\t\t\t\tif( el.readyState > 1 ) {\n\t\t\t\t\t\tthis.startEmbeddedMedia( { target: el } );\n\t\t\t\t\t}\n\t\t\t\t\t// Mobile devices never fire a loaded event so instead\n\t\t\t\t\t// of waiting, we initiate playback\n\t\t\t\t\telse if( isMobile ) {\n\t\t\t\t\t\tlet promise = el.play();\n\n\t\t\t\t\t\t// If autoplay does not work, ensure that the controls are visible so\n\t\t\t\t\t\t// that the viewer can start the media on their own\n\t\t\t\t\t\tif( promise && typeof promise.catch === 'function' && el.controls === false ) {\n\t\t\t\t\t\t\tpromise.catch( () => {\n\t\t\t\t\t\t\t\tel.controls = true;\n\n\t\t\t\t\t\t\t\t// Once the video does start playing, hide the controls again\n\t\t\t\t\t\t\t\tel.addEventListener( 'play', () => {\n\t\t\t\t\t\t\t\t\tel.controls = false;\n\t\t\t\t\t\t\t\t} );\n\t\t\t\t\t\t\t} );\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t// If the media isn't loaded, wait before playing\n\t\t\t\t\telse {\n\t\t\t\t\t\tel.removeEventListener( 'loadeddata', this.startEmbeddedMedia ); // remove first to avoid dupes\n\t\t\t\t\t\tel.addEventListener( 'loadeddata', this.startEmbeddedMedia );\n\t\t\t\t\t}\n\n\t\t\t\t}\n\t\t\t} );\n\n\t\t\t// Normal iframes\n\t\t\tqueryAll( element, 'iframe[src]' ).forEach( el => {\n\t\t\t\tif( closest( el, '.fragment' ) && !closest( el, '.fragment.visible' ) ) {\n\t\t\t\t\treturn;\n\t\t\t\t}\n\n\t\t\t\tthis.startEmbeddedIframe( { target: el } );\n\t\t\t} );\n\n\t\t\t// Lazy loading iframes\n\t\t\tqueryAll( element, 'iframe[data-src]' ).forEach( el => {\n\t\t\t\tif( closest( el, '.fragment' ) && !closest( el, '.fragment.visible' ) ) {\n\t\t\t\t\treturn;\n\t\t\t\t}\n\n\t\t\t\tif( el.getAttribute( 'src' ) !== el.getAttribute( 'data-src' ) ) {\n\t\t\t\t\tel.removeEventListener( 'load', this.startEmbeddedIframe ); // remove first to avoid dupes\n\t\t\t\t\tel.addEventListener( 'load', this.startEmbeddedIframe );\n\t\t\t\t\tel.setAttribute( 'src', el.getAttribute( 'data-src' ) );\n\t\t\t\t}\n\t\t\t} );\n\n\t\t}\n\n\t}\n\n\t/**\n\t * Starts playing an embedded video/audio element after\n\t * it has finished loading.\n\t *\n\t * @param {object} event\n\t */\n\tstartEmbeddedMedia( event ) {\n\n\t\tlet isAttachedToDOM = !!closest( event.target, 'html' ),\n\t\t\tisVisible \t\t= !!closest( event.target, '.present' );\n\n\t\tif( isAttachedToDOM && isVisible ) {\n\t\t\tevent.target.currentTime = 0;\n\t\t\tevent.target.play();\n\t\t}\n\n\t\tevent.target.removeEventListener( 'loadeddata', this.startEmbeddedMedia );\n\n\t}\n\n\t/**\n\t * \"Starts\" the content of an embedded iframe using the\n\t * postMessage API.\n\t *\n\t * @param {object} event\n\t */\n\tstartEmbeddedIframe( event ) {\n\n\t\tlet iframe = event.target;\n\n\t\tif( iframe && iframe.contentWindow ) {\n\n\t\t\tlet isAttachedToDOM = !!closest( event.target, 'html' ),\n\t\t\t\tisVisible \t\t= !!closest( event.target, '.present' );\n\n\t\t\tif( isAttachedToDOM && isVisible ) {\n\n\t\t\t\t// Prefer an explicit global autoplay setting\n\t\t\t\tlet autoplay = this.Reveal.getConfig().autoPlayMedia;\n\n\t\t\t\t// If no global setting is available, fall back on the element's\n\t\t\t\t// own autoplay setting\n\t\t\t\tif( typeof autoplay !== 'boolean' ) {\n\t\t\t\t\tautoplay = iframe.hasAttribute( 'data-autoplay' ) || !!closest( iframe, '.slide-background' );\n\t\t\t\t}\n\n\t\t\t\t// YouTube postMessage API\n\t\t\t\tif( /youtube\\.com\\/embed\\//.test( iframe.getAttribute( 'src' ) ) && autoplay ) {\n\t\t\t\t\tiframe.contentWindow.postMessage( '{\"event\":\"command\",\"func\":\"playVideo\",\"args\":\"\"}', '*' );\n\t\t\t\t}\n\t\t\t\t// Vimeo postMessage API\n\t\t\t\telse if( /player\\.vimeo\\.com\\//.test( iframe.getAttribute( 'src' ) ) && autoplay ) {\n\t\t\t\t\tiframe.contentWindow.postMessage( '{\"method\":\"play\"}', '*' );\n\t\t\t\t}\n\t\t\t\t// Generic postMessage API\n\t\t\t\telse {\n\t\t\t\t\tiframe.contentWindow.postMessage( 'slide:start', '*' );\n\t\t\t\t}\n\n\t\t\t}\n\n\t\t}\n\n\t}\n\n\t/**\n\t * Stop playback of any embedded content inside of\n\t * the targeted slide.\n\t *\n\t * @param {HTMLElement} element\n\t */\n\tstopEmbeddedContent( element, options = {} ) {\n\n\t\toptions = extend( {\n\t\t\t// Defaults\n\t\t\tunloadIframes: true\n\t\t}, options );\n\n\t\tif( element && element.parentNode ) {\n\t\t\t// HTML5 media elements\n\t\t\tqueryAll( element, 'video, audio' ).forEach( el => {\n\t\t\t\tif( !el.hasAttribute( 'data-ignore' ) && typeof el.pause === 'function' ) {\n\t\t\t\t\tel.setAttribute('data-paused-by-reveal', '');\n\t\t\t\t\tel.pause();\n\t\t\t\t}\n\t\t\t} );\n\n\t\t\t// Generic postMessage API for non-lazy loaded iframes\n\t\t\tqueryAll( element, 'iframe' ).forEach( el => {\n\t\t\t\tif( el.contentWindow ) el.contentWindow.postMessage( 'slide:stop', '*' );\n\t\t\t\tel.removeEventListener( 'load', this.startEmbeddedIframe );\n\t\t\t});\n\n\t\t\t// YouTube postMessage API\n\t\t\tqueryAll( element, 'iframe[src*=\"youtube.com/embed/\"]' ).forEach( el => {\n\t\t\t\tif( !el.hasAttribute( 'data-ignore' ) && el.contentWindow && typeof el.contentWindow.postMessage === 'function' ) {\n\t\t\t\t\tel.contentWindow.postMessage( '{\"event\":\"command\",\"func\":\"pauseVideo\",\"args\":\"\"}', '*' );\n\t\t\t\t}\n\t\t\t});\n\n\t\t\t// Vimeo postMessage API\n\t\t\tqueryAll( element, 'iframe[src*=\"player.vimeo.com/\"]' ).forEach( el => {\n\t\t\t\tif( !el.hasAttribute( 'data-ignore' ) && el.contentWindow && typeof el.contentWindow.postMessage === 'function' ) {\n\t\t\t\t\tel.contentWindow.postMessage( '{\"method\":\"pause\"}', '*' );\n\t\t\t\t}\n\t\t\t});\n\n\t\t\tif( options.unloadIframes === true ) {\n\t\t\t\t// Unload lazy-loaded iframes\n\t\t\t\tqueryAll( element, 'iframe[data-src]' ).forEach( el => {\n\t\t\t\t\t// Only removing the src doesn't actually unload the frame\n\t\t\t\t\t// in all browsers (Firefox) so we set it to blank first\n\t\t\t\t\tel.setAttribute( 'src', 'about:blank' );\n\t\t\t\t\tel.removeAttribute( 'src' );\n\t\t\t\t} );\n\t\t\t}\n\t\t}\n\n\t}\n\n}\n","/**\n * Handles the display of reveal.js' optional slide number.\n */\nexport default class SlideNumber {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t}\n\n\trender() {\n\n\t\tthis.element = document.createElement( 'div' );\n\t\tthis.element.className = 'slide-number';\n\t\tthis.Reveal.getRevealElement().appendChild( this.element );\n\n\t}\n\n\t/**\n\t * Called when the reveal.js config is updated.\n\t */\n\tconfigure( config, oldConfig ) {\n\n\t\tlet slideNumberDisplay = 'none';\n\t\tif( config.slideNumber && !this.Reveal.isPrintingPDF() ) {\n\t\t\tif( config.showSlideNumber === 'all' ) {\n\t\t\t\tslideNumberDisplay = 'block';\n\t\t\t}\n\t\t\telse if( config.showSlideNumber === 'speaker' && this.Reveal.isSpeakerNotes() ) {\n\t\t\t\tslideNumberDisplay = 'block';\n\t\t\t}\n\t\t}\n\n\t\tthis.element.style.display = slideNumberDisplay;\n\n\t}\n\n\t/**\n\t * Updates the slide number to match the current slide.\n\t */\n\tupdate() {\n\n\t\t// Update slide number if enabled\n\t\tif( this.Reveal.getConfig().slideNumber && this.element ) {\n\t\t\tthis.element.innerHTML = this.getSlideNumber();\n\t\t}\n\n\t}\n\n\t/**\n\t * Returns the HTML string corresponding to the current slide\n\t * number, including formatting.\n\t */\n\tgetSlideNumber( slide = this.Reveal.getCurrentSlide() ) {\n\n\t\tlet config = this.Reveal.getConfig();\n\t\tlet value;\n\t\tlet format = 'h.v';\n\n\t\tif ( typeof config.slideNumber === 'function' ) {\n\t\t\tvalue = config.slideNumber( slide );\n\t\t} else {\n\t\t\t// Check if a custom number format is available\n\t\t\tif( typeof config.slideNumber === 'string' ) {\n\t\t\t\tformat = config.slideNumber;\n\t\t\t}\n\n\t\t\t// If there are ONLY vertical slides in this deck, always use\n\t\t\t// a flattened slide number\n\t\t\tif( !/c/.test( format ) && this.Reveal.getHorizontalSlides().length === 1 ) {\n\t\t\t\tformat = 'c';\n\t\t\t}\n\n\t\t\t// Offset the current slide number by 1 to make it 1-indexed\n\t\t\tlet horizontalOffset = slide && slide.dataset.visibility === 'uncounted' ? 0 : 1;\n\n\t\t\tvalue = [];\n\t\t\tswitch( format ) {\n\t\t\t\tcase 'c':\n\t\t\t\t\tvalue.push( this.Reveal.getSlidePastCount( slide ) + horizontalOffset );\n\t\t\t\t\tbreak;\n\t\t\t\tcase 'c/t':\n\t\t\t\t\tvalue.push( this.Reveal.getSlidePastCount( slide ) + horizontalOffset, '/', this.Reveal.getTotalSlides() );\n\t\t\t\t\tbreak;\n\t\t\t\tdefault:\n\t\t\t\t\tlet indices = this.Reveal.getIndices( slide );\n\t\t\t\t\tvalue.push( indices.h + horizontalOffset );\n\t\t\t\t\tlet sep = format === 'h/v' ? '/' : '.';\n\t\t\t\t\tif( this.Reveal.isVerticalSlide( slide ) ) value.push( sep, indices.v + 1 );\n\t\t\t}\n\t\t}\n\n\t\tlet url = '#' + this.Reveal.location.getHash( slide );\n\t\treturn this.formatNumber( value[0], value[1], value[2], url );\n\n\t}\n\n\t/**\n\t * Applies HTML formatting to a slide number before it's\n\t * written to the DOM.\n\t *\n\t * @param {number} a Current slide\n\t * @param {string} delimiter Character to separate slide numbers\n\t * @param {(number|*)} b Total slides\n\t * @param {HTMLElement} [url='#'+locationHash()] The url to link to\n\t * @return {string} HTML string fragment\n\t */\n\tformatNumber( a, delimiter, b, url = '#' + this.Reveal.location.getHash() ) {\n\n\t\tif( typeof b === 'number' && !isNaN( b ) ) {\n\t\t\treturn `\n\t\t\t\t\t${a}\n\t\t\t\t\t${delimiter}\n\t\t\t\t\t${b}\n\t\t\t\t\t`;\n\t\t}\n\t\telse {\n\t\t\treturn `\n\t\t\t\t\t${a}\n\t\t\t\t\t`;\n\t\t}\n\n\t}\n\n\tdestroy() {\n\n\t\tthis.element.remove();\n\n\t}\n\n}","/**\n * Converts various color input formats to an {r:0,g:0,b:0} object.\n *\n * @param {string} color The string representation of a color\n * @example\n * colorToRgb('#000');\n * @example\n * colorToRgb('#000000');\n * @example\n * colorToRgb('rgb(0,0,0)');\n * @example\n * colorToRgb('rgba(0,0,0)');\n *\n * @return {{r: number, g: number, b: number, [a]: number}|null}\n */\nexport const colorToRgb = ( color ) => {\n\n\tlet hex3 = color.match( /^#([0-9a-f]{3})$/i );\n\tif( hex3 && hex3[1] ) {\n\t\thex3 = hex3[1];\n\t\treturn {\n\t\t\tr: parseInt( hex3.charAt( 0 ), 16 ) * 0x11,\n\t\t\tg: parseInt( hex3.charAt( 1 ), 16 ) * 0x11,\n\t\t\tb: parseInt( hex3.charAt( 2 ), 16 ) * 0x11\n\t\t};\n\t}\n\n\tlet hex6 = color.match( /^#([0-9a-f]{6})$/i );\n\tif( hex6 && hex6[1] ) {\n\t\thex6 = hex6[1];\n\t\treturn {\n\t\t\tr: parseInt( hex6.slice( 0, 2 ), 16 ),\n\t\t\tg: parseInt( hex6.slice( 2, 4 ), 16 ),\n\t\t\tb: parseInt( hex6.slice( 4, 6 ), 16 )\n\t\t};\n\t}\n\n\tlet rgb = color.match( /^rgb\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\)$/i );\n\tif( rgb ) {\n\t\treturn {\n\t\t\tr: parseInt( rgb[1], 10 ),\n\t\t\tg: parseInt( rgb[2], 10 ),\n\t\t\tb: parseInt( rgb[3], 10 )\n\t\t};\n\t}\n\n\tlet rgba = color.match( /^rgba\\s*\\(\\s*(\\d+)\\s*,\\s*(\\d+)\\s*,\\s*(\\d+)\\s*\\,\\s*([\\d]+|[\\d]*.[\\d]+)\\s*\\)$/i );\n\tif( rgba ) {\n\t\treturn {\n\t\t\tr: parseInt( rgba[1], 10 ),\n\t\t\tg: parseInt( rgba[2], 10 ),\n\t\t\tb: parseInt( rgba[3], 10 ),\n\t\t\ta: parseFloat( rgba[4] )\n\t\t};\n\t}\n\n\treturn null;\n\n}\n\n/**\n * Calculates brightness on a scale of 0-255.\n *\n * @param {string} color See colorToRgb for supported formats.\n * @see {@link colorToRgb}\n */\nexport const colorBrightness = ( color ) => {\n\n\tif( typeof color === 'string' ) color = colorToRgb( color );\n\n\tif( color ) {\n\t\treturn ( color.r * 299 + color.g * 587 + color.b * 114 ) / 1000;\n\t}\n\n\treturn null;\n\n}","import { queryAll } from '../utils/util.js'\nimport { colorToRgb, colorBrightness } from '../utils/color.js'\n\n/**\n * Creates and updates slide backgrounds.\n */\nexport default class Backgrounds {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t}\n\n\trender() {\n\n\t\tthis.element = document.createElement( 'div' );\n\t\tthis.element.className = 'backgrounds';\n\t\tthis.Reveal.getRevealElement().appendChild( this.element );\n\n\t}\n\n\t/**\n\t * Creates the slide background elements and appends them\n\t * to the background container. One element is created per\n\t * slide no matter if the given slide has visible background.\n\t */\n\tcreate() {\n\n\t\t// Clear prior backgrounds\n\t\tthis.element.innerHTML = '';\n\t\tthis.element.classList.add( 'no-transition' );\n\n\t\t// Iterate over all horizontal slides\n\t\tthis.Reveal.getHorizontalSlides().forEach( slideh => {\n\n\t\t\tlet backgroundStack = this.createBackground( slideh, this.element );\n\n\t\t\t// Iterate over all vertical slides\n\t\t\tqueryAll( slideh, 'section' ).forEach( slidev => {\n\n\t\t\t\tthis.createBackground( slidev, backgroundStack );\n\n\t\t\t\tbackgroundStack.classList.add( 'stack' );\n\n\t\t\t} );\n\n\t\t} );\n\n\t\t// Add parallax background if specified\n\t\tif( this.Reveal.getConfig().parallaxBackgroundImage ) {\n\n\t\t\tthis.element.style.backgroundImage = 'url(\"' + this.Reveal.getConfig().parallaxBackgroundImage + '\")';\n\t\t\tthis.element.style.backgroundSize = this.Reveal.getConfig().parallaxBackgroundSize;\n\t\t\tthis.element.style.backgroundRepeat = this.Reveal.getConfig().parallaxBackgroundRepeat;\n\t\t\tthis.element.style.backgroundPosition = this.Reveal.getConfig().parallaxBackgroundPosition;\n\n\t\t\t// Make sure the below properties are set on the element - these properties are\n\t\t\t// needed for proper transitions to be set on the element via CSS. To remove\n\t\t\t// annoying background slide-in effect when the presentation starts, apply\n\t\t\t// these properties after short time delay\n\t\t\tsetTimeout( () => {\n\t\t\t\tthis.Reveal.getRevealElement().classList.add( 'has-parallax-background' );\n\t\t\t}, 1 );\n\n\t\t}\n\t\telse {\n\n\t\t\tthis.element.style.backgroundImage = '';\n\t\t\tthis.Reveal.getRevealElement().classList.remove( 'has-parallax-background' );\n\n\t\t}\n\n\t}\n\n\t/**\n\t * Creates a background for the given slide.\n\t *\n\t * @param {HTMLElement} slide\n\t * @param {HTMLElement} container The element that the background\n\t * should be appended to\n\t * @return {HTMLElement} New background div\n\t */\n\tcreateBackground( slide, container ) {\n\n\t\t// Main slide background element\n\t\tlet element = document.createElement( 'div' );\n\t\telement.className = 'slide-background ' + slide.className.replace( /present|past|future/, '' );\n\n\t\t// Inner background element that wraps images/videos/iframes\n\t\tlet contentElement = document.createElement( 'div' );\n\t\tcontentElement.className = 'slide-background-content';\n\n\t\telement.appendChild( contentElement );\n\t\tcontainer.appendChild( element );\n\n\t\tslide.slideBackgroundElement = element;\n\t\tslide.slideBackgroundContentElement = contentElement;\n\n\t\t// Syncs the background to reflect all current background settings\n\t\tthis.sync( slide );\n\n\t\treturn element;\n\n\t}\n\n\t/**\n\t * Renders all of the visual properties of a slide background\n\t * based on the various background attributes.\n\t *\n\t * @param {HTMLElement} slide\n\t */\n\tsync( slide ) {\n\n\t\tconst element = slide.slideBackgroundElement,\n\t\t\tcontentElement = slide.slideBackgroundContentElement;\n\n\t\tconst data = {\n\t\t\tbackground: slide.getAttribute( 'data-background' ),\n\t\t\tbackgroundSize: slide.getAttribute( 'data-background-size' ),\n\t\t\tbackgroundImage: slide.getAttribute( 'data-background-image' ),\n\t\t\tbackgroundVideo: slide.getAttribute( 'data-background-video' ),\n\t\t\tbackgroundIframe: slide.getAttribute( 'data-background-iframe' ),\n\t\t\tbackgroundColor: slide.getAttribute( 'data-background-color' ),\n\t\t\tbackgroundRepeat: slide.getAttribute( 'data-background-repeat' ),\n\t\t\tbackgroundPosition: slide.getAttribute( 'data-background-position' ),\n\t\t\tbackgroundTransition: slide.getAttribute( 'data-background-transition' ),\n\t\t\tbackgroundOpacity: slide.getAttribute( 'data-background-opacity' ),\n\t\t};\n\n\t\tconst dataPreload = slide.hasAttribute( 'data-preload' );\n\n\t\t// Reset the prior background state in case this is not the\n\t\t// initial sync\n\t\tslide.classList.remove( 'has-dark-background' );\n\t\tslide.classList.remove( 'has-light-background' );\n\n\t\telement.removeAttribute( 'data-loaded' );\n\t\telement.removeAttribute( 'data-background-hash' );\n\t\telement.removeAttribute( 'data-background-size' );\n\t\telement.removeAttribute( 'data-background-transition' );\n\t\telement.style.backgroundColor = '';\n\n\t\tcontentElement.style.backgroundSize = '';\n\t\tcontentElement.style.backgroundRepeat = '';\n\t\tcontentElement.style.backgroundPosition = '';\n\t\tcontentElement.style.backgroundImage = '';\n\t\tcontentElement.style.opacity = '';\n\t\tcontentElement.innerHTML = '';\n\n\t\tif( data.background ) {\n\t\t\t// Auto-wrap image urls in url(...)\n\t\t\tif( /^(http|file|\\/\\/)/gi.test( data.background ) || /\\.(svg|png|jpg|jpeg|gif|bmp)([?#\\s]|$)/gi.test( data.background ) ) {\n\t\t\t\tslide.setAttribute( 'data-background-image', data.background );\n\t\t\t}\n\t\t\telse {\n\t\t\t\telement.style.background = data.background;\n\t\t\t}\n\t\t}\n\n\t\t// Create a hash for this combination of background settings.\n\t\t// This is used to determine when two slide backgrounds are\n\t\t// the same.\n\t\tif( data.background || data.backgroundColor || data.backgroundImage || data.backgroundVideo || data.backgroundIframe ) {\n\t\t\telement.setAttribute( 'data-background-hash', data.background +\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tdata.backgroundSize +\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tdata.backgroundImage +\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tdata.backgroundVideo +\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tdata.backgroundIframe +\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tdata.backgroundColor +\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tdata.backgroundRepeat +\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tdata.backgroundPosition +\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tdata.backgroundTransition +\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tdata.backgroundOpacity );\n\t\t}\n\n\t\t// Additional and optional background properties\n\t\tif( data.backgroundSize ) element.setAttribute( 'data-background-size', data.backgroundSize );\n\t\tif( data.backgroundColor ) element.style.backgroundColor = data.backgroundColor;\n\t\tif( data.backgroundTransition ) element.setAttribute( 'data-background-transition', data.backgroundTransition );\n\n\t\tif( dataPreload ) element.setAttribute( 'data-preload', '' );\n\n\t\t// Background image options are set on the content wrapper\n\t\tif( data.backgroundSize ) contentElement.style.backgroundSize = data.backgroundSize;\n\t\tif( data.backgroundRepeat ) contentElement.style.backgroundRepeat = data.backgroundRepeat;\n\t\tif( data.backgroundPosition ) contentElement.style.backgroundPosition = data.backgroundPosition;\n\t\tif( data.backgroundOpacity ) contentElement.style.opacity = data.backgroundOpacity;\n\n\t\t// If this slide has a background color, we add a class that\n\t\t// signals if it is light or dark. If the slide has no background\n\t\t// color, no class will be added\n\t\tlet contrastColor = data.backgroundColor;\n\n\t\t// If no bg color was found, or it cannot be converted by colorToRgb, check the computed background\n\t\tif( !contrastColor || !colorToRgb( contrastColor ) ) {\n\t\t\tlet computedBackgroundStyle = window.getComputedStyle( element );\n\t\t\tif( computedBackgroundStyle && computedBackgroundStyle.backgroundColor ) {\n\t\t\t\tcontrastColor = computedBackgroundStyle.backgroundColor;\n\t\t\t}\n\t\t}\n\n\t\tif( contrastColor ) {\n\t\t\tconst rgb = colorToRgb( contrastColor );\n\n\t\t\t// Ignore fully transparent backgrounds. Some browsers return\n\t\t\t// rgba(0,0,0,0) when reading the computed background color of\n\t\t\t// an element with no background\n\t\t\tif( rgb && rgb.a !== 0 ) {\n\t\t\t\tif( colorBrightness( contrastColor ) < 128 ) {\n\t\t\t\t\tslide.classList.add( 'has-dark-background' );\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tslide.classList.add( 'has-light-background' );\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t}\n\n\t/**\n\t * Updates the background elements to reflect the current\n\t * slide.\n\t *\n\t * @param {boolean} includeAll If true, the backgrounds of\n\t * all vertical slides (not just the present) will be updated.\n\t */\n\tupdate( includeAll = false ) {\n\n\t\tlet currentSlide = this.Reveal.getCurrentSlide();\n\t\tlet indices = this.Reveal.getIndices();\n\n\t\tlet currentBackground = null;\n\n\t\t// Reverse past/future classes when in RTL mode\n\t\tlet horizontalPast = this.Reveal.getConfig().rtl ? 'future' : 'past',\n\t\t\thorizontalFuture = this.Reveal.getConfig().rtl ? 'past' : 'future';\n\n\t\t// Update the classes of all backgrounds to match the\n\t\t// states of their slides (past/present/future)\n\t\tArray.from( this.element.childNodes ).forEach( ( backgroundh, h ) => {\n\n\t\t\tbackgroundh.classList.remove( 'past', 'present', 'future' );\n\n\t\t\tif( h < indices.h ) {\n\t\t\t\tbackgroundh.classList.add( horizontalPast );\n\t\t\t}\n\t\t\telse if ( h > indices.h ) {\n\t\t\t\tbackgroundh.classList.add( horizontalFuture );\n\t\t\t}\n\t\t\telse {\n\t\t\t\tbackgroundh.classList.add( 'present' );\n\n\t\t\t\t// Store a reference to the current background element\n\t\t\t\tcurrentBackground = backgroundh;\n\t\t\t}\n\n\t\t\tif( includeAll || h === indices.h ) {\n\t\t\t\tqueryAll( backgroundh, '.slide-background' ).forEach( ( backgroundv, v ) => {\n\n\t\t\t\t\tbackgroundv.classList.remove( 'past', 'present', 'future' );\n\n\t\t\t\t\tif( v < indices.v ) {\n\t\t\t\t\t\tbackgroundv.classList.add( 'past' );\n\t\t\t\t\t}\n\t\t\t\t\telse if ( v > indices.v ) {\n\t\t\t\t\t\tbackgroundv.classList.add( 'future' );\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tbackgroundv.classList.add( 'present' );\n\n\t\t\t\t\t\t// Only if this is the present horizontal and vertical slide\n\t\t\t\t\t\tif( h === indices.h ) currentBackground = backgroundv;\n\t\t\t\t\t}\n\n\t\t\t\t} );\n\t\t\t}\n\n\t\t} );\n\n\t\t// Stop content inside of previous backgrounds\n\t\tif( this.previousBackground ) {\n\n\t\t\tthis.Reveal.slideContent.stopEmbeddedContent( this.previousBackground, { unloadIframes: !this.Reveal.slideContent.shouldPreload( this.previousBackground ) } );\n\n\t\t}\n\n\t\t// Start content in the current background\n\t\tif( currentBackground ) {\n\n\t\t\tthis.Reveal.slideContent.startEmbeddedContent( currentBackground );\n\n\t\t\tlet currentBackgroundContent = currentBackground.querySelector( '.slide-background-content' );\n\t\t\tif( currentBackgroundContent ) {\n\n\t\t\t\tlet backgroundImageURL = currentBackgroundContent.style.backgroundImage || '';\n\n\t\t\t\t// Restart GIFs (doesn't work in Firefox)\n\t\t\t\tif( /\\.gif/i.test( backgroundImageURL ) ) {\n\t\t\t\t\tcurrentBackgroundContent.style.backgroundImage = '';\n\t\t\t\t\twindow.getComputedStyle( currentBackgroundContent ).opacity;\n\t\t\t\t\tcurrentBackgroundContent.style.backgroundImage = backgroundImageURL;\n\t\t\t\t}\n\n\t\t\t}\n\n\t\t\t// Don't transition between identical backgrounds. This\n\t\t\t// prevents unwanted flicker.\n\t\t\tlet previousBackgroundHash = this.previousBackground ? this.previousBackground.getAttribute( 'data-background-hash' ) : null;\n\t\t\tlet currentBackgroundHash = currentBackground.getAttribute( 'data-background-hash' );\n\t\t\tif( currentBackgroundHash && currentBackgroundHash === previousBackgroundHash && currentBackground !== this.previousBackground ) {\n\t\t\t\tthis.element.classList.add( 'no-transition' );\n\t\t\t}\n\n\t\t\tthis.previousBackground = currentBackground;\n\n\t\t}\n\n\t\t// If there's a background brightness flag for this slide,\n\t\t// bubble it to the .reveal container\n\t\tif( currentSlide ) {\n\t\t\t[ 'has-light-background', 'has-dark-background' ].forEach( classToBubble => {\n\t\t\t\tif( currentSlide.classList.contains( classToBubble ) ) {\n\t\t\t\t\tthis.Reveal.getRevealElement().classList.add( classToBubble );\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tthis.Reveal.getRevealElement().classList.remove( classToBubble );\n\t\t\t\t}\n\t\t\t}, this );\n\t\t}\n\n\t\t// Allow the first background to apply without transition\n\t\tsetTimeout( () => {\n\t\t\tthis.element.classList.remove( 'no-transition' );\n\t\t}, 1 );\n\n\t}\n\n\t/**\n\t * Updates the position of the parallax background based\n\t * on the current slide index.\n\t */\n\tupdateParallax() {\n\n\t\tlet indices = this.Reveal.getIndices();\n\n\t\tif( this.Reveal.getConfig().parallaxBackgroundImage ) {\n\n\t\t\tlet horizontalSlides = this.Reveal.getHorizontalSlides(),\n\t\t\t\tverticalSlides = this.Reveal.getVerticalSlides();\n\n\t\t\tlet backgroundSize = this.element.style.backgroundSize.split( ' ' ),\n\t\t\t\tbackgroundWidth, backgroundHeight;\n\n\t\t\tif( backgroundSize.length === 1 ) {\n\t\t\t\tbackgroundWidth = backgroundHeight = parseInt( backgroundSize[0], 10 );\n\t\t\t}\n\t\t\telse {\n\t\t\t\tbackgroundWidth = parseInt( backgroundSize[0], 10 );\n\t\t\t\tbackgroundHeight = parseInt( backgroundSize[1], 10 );\n\t\t\t}\n\n\t\t\tlet slideWidth = this.element.offsetWidth,\n\t\t\t\thorizontalSlideCount = horizontalSlides.length,\n\t\t\t\thorizontalOffsetMultiplier,\n\t\t\t\thorizontalOffset;\n\n\t\t\tif( typeof this.Reveal.getConfig().parallaxBackgroundHorizontal === 'number' ) {\n\t\t\t\thorizontalOffsetMultiplier = this.Reveal.getConfig().parallaxBackgroundHorizontal;\n\t\t\t}\n\t\t\telse {\n\t\t\t\thorizontalOffsetMultiplier = horizontalSlideCount > 1 ? ( backgroundWidth - slideWidth ) / ( horizontalSlideCount-1 ) : 0;\n\t\t\t}\n\n\t\t\thorizontalOffset = horizontalOffsetMultiplier * indices.h * -1;\n\n\t\t\tlet slideHeight = this.element.offsetHeight,\n\t\t\t\tverticalSlideCount = verticalSlides.length,\n\t\t\t\tverticalOffsetMultiplier,\n\t\t\t\tverticalOffset;\n\n\t\t\tif( typeof this.Reveal.getConfig().parallaxBackgroundVertical === 'number' ) {\n\t\t\t\tverticalOffsetMultiplier = this.Reveal.getConfig().parallaxBackgroundVertical;\n\t\t\t}\n\t\t\telse {\n\t\t\t\tverticalOffsetMultiplier = ( backgroundHeight - slideHeight ) / ( verticalSlideCount-1 );\n\t\t\t}\n\n\t\t\tverticalOffset = verticalSlideCount > 0 ? verticalOffsetMultiplier * indices.v : 0;\n\n\t\t\tthis.element.style.backgroundPosition = horizontalOffset + 'px ' + -verticalOffset + 'px';\n\n\t\t}\n\n\t}\n\n\tdestroy() {\n\n\t\tthis.element.remove();\n\n\t}\n\n}\n","\nexport const SLIDES_SELECTOR = '.slides section';\nexport const HORIZONTAL_SLIDES_SELECTOR = '.slides>section';\nexport const VERTICAL_SLIDES_SELECTOR = '.slides>section.present>section';\n\n// Methods that may not be invoked via the postMessage API\nexport const POST_MESSAGE_METHOD_BLACKLIST = /registerPlugin|registerKeyboardShortcut|addKeyBinding|addEventListener/;\n\n// Regex for retrieving the fragment style from a class attribute\nexport const FRAGMENT_STYLE_REGEX = /fade-(down|up|right|left|out|in-then-out|in-then-semi-out)|semi-fade-out|current-visible|shrink|grow/;","import { queryAll, extend, createStyleSheet, matches, closest } from '../utils/util.js'\nimport { FRAGMENT_STYLE_REGEX } from '../utils/constants.js'\n\n// Counter used to generate unique IDs for auto-animated elements\nlet autoAnimateCounter = 0;\n\n/**\n * Automatically animates matching elements across\n * slides with the [data-auto-animate] attribute.\n */\nexport default class AutoAnimate {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t}\n\n\t/**\n\t * Runs an auto-animation between the given slides.\n\t *\n\t * @param {HTMLElement} fromSlide\n\t * @param {HTMLElement} toSlide\n\t */\n\trun( fromSlide, toSlide ) {\n\n\t\t// Clean up after prior animations\n\t\tthis.reset();\n\n\t\tlet allSlides = this.Reveal.getSlides();\n\t\tlet toSlideIndex = allSlides.indexOf( toSlide );\n\t\tlet fromSlideIndex = allSlides.indexOf( fromSlide );\n\n\t\t// Ensure that both slides are auto-animate targets with the same data-auto-animate-id value\n\t\t// (including null if absent on both) and that data-auto-animate-restart isn't set on the\n\t\t// physically latter slide (independent of slide direction)\n\t\tif( fromSlide.hasAttribute( 'data-auto-animate' ) && toSlide.hasAttribute( 'data-auto-animate' )\n\t\t\t\t&& fromSlide.getAttribute( 'data-auto-animate-id' ) === toSlide.getAttribute( 'data-auto-animate-id' ) \n\t\t\t\t&& !( toSlideIndex > fromSlideIndex ? toSlide : fromSlide ).hasAttribute( 'data-auto-animate-restart' ) ) {\n\n\t\t\t// Create a new auto-animate sheet\n\t\t\tthis.autoAnimateStyleSheet = this.autoAnimateStyleSheet || createStyleSheet();\n\n\t\t\tlet animationOptions = this.getAutoAnimateOptions( toSlide );\n\n\t\t\t// Set our starting state\n\t\t\tfromSlide.dataset.autoAnimate = 'pending';\n\t\t\ttoSlide.dataset.autoAnimate = 'pending';\n\n\t\t\t// Flag the navigation direction, needed for fragment buildup\n\t\t\tanimationOptions.slideDirection = toSlideIndex > fromSlideIndex ? 'forward' : 'backward';\n\n\t\t\t// Inject our auto-animate styles for this transition\n\t\t\tlet css = this.getAutoAnimatableElements( fromSlide, toSlide ).map( elements => {\n\t\t\t\treturn this.autoAnimateElements( elements.from, elements.to, elements.options || {}, animationOptions, autoAnimateCounter++ );\n\t\t\t} );\n\n\t\t\t// Animate unmatched elements, if enabled\n\t\t\tif( toSlide.dataset.autoAnimateUnmatched !== 'false' && this.Reveal.getConfig().autoAnimateUnmatched === true ) {\n\n\t\t\t\t// Our default timings for unmatched elements\n\t\t\t\tlet defaultUnmatchedDuration = animationOptions.duration * 0.8,\n\t\t\t\t\tdefaultUnmatchedDelay = animationOptions.duration * 0.2;\n\n\t\t\t\tthis.getUnmatchedAutoAnimateElements( toSlide ).forEach( unmatchedElement => {\n\n\t\t\t\t\tlet unmatchedOptions = this.getAutoAnimateOptions( unmatchedElement, animationOptions );\n\t\t\t\t\tlet id = 'unmatched';\n\n\t\t\t\t\t// If there is a duration or delay set specifically for this\n\t\t\t\t\t// element our unmatched elements should adhere to those\n\t\t\t\t\tif( unmatchedOptions.duration !== animationOptions.duration || unmatchedOptions.delay !== animationOptions.delay ) {\n\t\t\t\t\t\tid = 'unmatched-' + autoAnimateCounter++;\n\t\t\t\t\t\tcss.push( `[data-auto-animate=\"running\"] [data-auto-animate-target=\"${id}\"] { transition: opacity ${unmatchedOptions.duration}s ease ${unmatchedOptions.delay}s; }` );\n\t\t\t\t\t}\n\n\t\t\t\t\tunmatchedElement.dataset.autoAnimateTarget = id;\n\n\t\t\t\t}, this );\n\n\t\t\t\t// Our default transition for unmatched elements\n\t\t\t\tcss.push( `[data-auto-animate=\"running\"] [data-auto-animate-target=\"unmatched\"] { transition: opacity ${defaultUnmatchedDuration}s ease ${defaultUnmatchedDelay}s; }` );\n\n\t\t\t}\n\n\t\t\t// Setting the whole chunk of CSS at once is the most\n\t\t\t// efficient way to do this. Using sheet.insertRule\n\t\t\t// is multiple factors slower.\n\t\t\tthis.autoAnimateStyleSheet.innerHTML = css.join( '' );\n\n\t\t\t// Start the animation next cycle\n\t\t\trequestAnimationFrame( () => {\n\t\t\t\tif( this.autoAnimateStyleSheet ) {\n\t\t\t\t\t// This forces our newly injected styles to be applied in Firefox\n\t\t\t\t\tgetComputedStyle( this.autoAnimateStyleSheet ).fontWeight;\n\n\t\t\t\t\ttoSlide.dataset.autoAnimate = 'running';\n\t\t\t\t}\n\t\t\t} );\n\n\t\t\tthis.Reveal.dispatchEvent({\n\t\t\t\ttype: 'autoanimate',\n\t\t\t\tdata: {\n\t\t\t\t\tfromSlide,\n\t\t\t\t\ttoSlide,\n\t\t\t\t\tsheet: this.autoAnimateStyleSheet\n\t\t\t\t}\n\t\t\t});\n\n\t\t}\n\n\t}\n\n\t/**\n\t * Rolls back all changes that we've made to the DOM so\n\t * that as part of animating.\n\t */\n\treset() {\n\n\t\t// Reset slides\n\t\tqueryAll( this.Reveal.getRevealElement(), '[data-auto-animate]:not([data-auto-animate=\"\"])' ).forEach( element => {\n\t\t\telement.dataset.autoAnimate = '';\n\t\t} );\n\n\t\t// Reset elements\n\t\tqueryAll( this.Reveal.getRevealElement(), '[data-auto-animate-target]' ).forEach( element => {\n\t\t\tdelete element.dataset.autoAnimateTarget;\n\t\t} );\n\n\t\t// Remove the animation sheet\n\t\tif( this.autoAnimateStyleSheet && this.autoAnimateStyleSheet.parentNode ) {\n\t\t\tthis.autoAnimateStyleSheet.parentNode.removeChild( this.autoAnimateStyleSheet );\n\t\t\tthis.autoAnimateStyleSheet = null;\n\t\t}\n\n\t}\n\n\t/**\n\t * Creates a FLIP animation where the `to` element starts out\n\t * in the `from` element position and animates to its original\n\t * state.\n\t *\n\t * @param {HTMLElement} from\n\t * @param {HTMLElement} to\n\t * @param {Object} elementOptions Options for this element pair\n\t * @param {Object} animationOptions Options set at the slide level\n\t * @param {String} id Unique ID that we can use to identify this\n\t * auto-animate element in the DOM\n\t */\n\tautoAnimateElements( from, to, elementOptions, animationOptions, id ) {\n\n\t\t// 'from' elements are given a data-auto-animate-target with no value,\n\t\t// 'to' elements are are given a data-auto-animate-target with an ID\n\t\tfrom.dataset.autoAnimateTarget = '';\n\t\tto.dataset.autoAnimateTarget = id;\n\n\t\t// Each element may override any of the auto-animate options\n\t\t// like transition easing, duration and delay via data-attributes\n\t\tlet options = this.getAutoAnimateOptions( to, animationOptions );\n\n\t\t// If we're using a custom element matcher the element options\n\t\t// may contain additional transition overrides\n\t\tif( typeof elementOptions.delay !== 'undefined' ) options.delay = elementOptions.delay;\n\t\tif( typeof elementOptions.duration !== 'undefined' ) options.duration = elementOptions.duration;\n\t\tif( typeof elementOptions.easing !== 'undefined' ) options.easing = elementOptions.easing;\n\n\t\tlet fromProps = this.getAutoAnimatableProperties( 'from', from, elementOptions ),\n\t\t\ttoProps = this.getAutoAnimatableProperties( 'to', to, elementOptions );\n\n\t\t// Maintain fragment visibility for matching elements when\n\t\t// we're navigating forwards, this way the viewer won't need\n\t\t// to step through the same fragments twice\n\t\tif( to.classList.contains( 'fragment' ) ) {\n\n\t\t\t// Don't auto-animate the opacity of fragments to avoid\n\t\t\t// conflicts with fragment animations\n\t\t\tdelete toProps.styles['opacity'];\n\n\t\t\tif( from.classList.contains( 'fragment' ) ) {\n\n\t\t\t\tlet fromFragmentStyle = ( from.className.match( FRAGMENT_STYLE_REGEX ) || [''] )[0];\n\t\t\t\tlet toFragmentStyle = ( to.className.match( FRAGMENT_STYLE_REGEX ) || [''] )[0];\n\n\t\t\t\t// Only skip the fragment if the fragment animation style\n\t\t\t\t// remains unchanged\n\t\t\t\tif( fromFragmentStyle === toFragmentStyle && animationOptions.slideDirection === 'forward' ) {\n\t\t\t\t\tto.classList.add( 'visible', 'disabled' );\n\t\t\t\t}\n\n\t\t\t}\n\n\t\t}\n\n\t\t// If translation and/or scaling are enabled, css transform\n\t\t// the 'to' element so that it matches the position and size\n\t\t// of the 'from' element\n\t\tif( elementOptions.translate !== false || elementOptions.scale !== false ) {\n\n\t\t\tlet presentationScale = this.Reveal.getScale();\n\n\t\t\tlet delta = {\n\t\t\t\tx: ( fromProps.x - toProps.x ) / presentationScale,\n\t\t\t\ty: ( fromProps.y - toProps.y ) / presentationScale,\n\t\t\t\tscaleX: fromProps.width / toProps.width,\n\t\t\t\tscaleY: fromProps.height / toProps.height\n\t\t\t};\n\n\t\t\t// Limit decimal points to avoid 0.0001px blur and stutter\n\t\t\tdelta.x = Math.round( delta.x * 1000 ) / 1000;\n\t\t\tdelta.y = Math.round( delta.y * 1000 ) / 1000;\n\t\t\tdelta.scaleX = Math.round( delta.scaleX * 1000 ) / 1000;\n\t\t\tdelta.scaleX = Math.round( delta.scaleX * 1000 ) / 1000;\n\n\t\t\tlet translate = elementOptions.translate !== false && ( delta.x !== 0 || delta.y !== 0 ),\n\t\t\t\tscale = elementOptions.scale !== false && ( delta.scaleX !== 0 || delta.scaleY !== 0 );\n\n\t\t\t// No need to transform if nothing's changed\n\t\t\tif( translate || scale ) {\n\n\t\t\t\tlet transform = [];\n\n\t\t\t\tif( translate ) transform.push( `translate(${delta.x}px, ${delta.y}px)` );\n\t\t\t\tif( scale ) transform.push( `scale(${delta.scaleX}, ${delta.scaleY})` );\n\n\t\t\t\tfromProps.styles['transform'] = transform.join( ' ' );\n\t\t\t\tfromProps.styles['transform-origin'] = 'top left';\n\n\t\t\t\ttoProps.styles['transform'] = 'none';\n\n\t\t\t}\n\n\t\t}\n\n\t\t// Delete all unchanged 'to' styles\n\t\tfor( let propertyName in toProps.styles ) {\n\t\t\tconst toValue = toProps.styles[propertyName];\n\t\t\tconst fromValue = fromProps.styles[propertyName];\n\n\t\t\tif( toValue === fromValue ) {\n\t\t\t\tdelete toProps.styles[propertyName];\n\t\t\t}\n\t\t\telse {\n\t\t\t\t// If these property values were set via a custom matcher providing\n\t\t\t\t// an explicit 'from' and/or 'to' value, we always inject those values.\n\t\t\t\tif( toValue.explicitValue === true ) {\n\t\t\t\t\ttoProps.styles[propertyName] = toValue.value;\n\t\t\t\t}\n\n\t\t\t\tif( fromValue.explicitValue === true ) {\n\t\t\t\t\tfromProps.styles[propertyName] = fromValue.value;\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\tlet css = '';\n\n\t\tlet toStyleProperties = Object.keys( toProps.styles );\n\n\t\t// Only create animate this element IF at least one style\n\t\t// property has changed\n\t\tif( toStyleProperties.length > 0 ) {\n\n\t\t\t// Instantly move to the 'from' state\n\t\t\tfromProps.styles['transition'] = 'none';\n\n\t\t\t// Animate towards the 'to' state\n\t\t\ttoProps.styles['transition'] = `all ${options.duration}s ${options.easing} ${options.delay}s`;\n\t\t\ttoProps.styles['transition-property'] = toStyleProperties.join( ', ' );\n\t\t\ttoProps.styles['will-change'] = toStyleProperties.join( ', ' );\n\n\t\t\t// Build up our custom CSS. We need to override inline styles\n\t\t\t// so we need to make our styles vErY IMPORTANT!1!!\n\t\t\tlet fromCSS = Object.keys( fromProps.styles ).map( propertyName => {\n\t\t\t\treturn propertyName + ': ' + fromProps.styles[propertyName] + ' !important;';\n\t\t\t} ).join( '' );\n\n\t\t\tlet toCSS = Object.keys( toProps.styles ).map( propertyName => {\n\t\t\t\treturn propertyName + ': ' + toProps.styles[propertyName] + ' !important;';\n\t\t\t} ).join( '' );\n\n\t\t\tcss = \t'[data-auto-animate-target=\"'+ id +'\"] {'+ fromCSS +'}' +\n\t\t\t\t\t'[data-auto-animate=\"running\"] [data-auto-animate-target=\"'+ id +'\"] {'+ toCSS +'}';\n\n\t\t}\n\n\t\treturn css;\n\n\t}\n\n\t/**\n\t * Returns the auto-animate options for the given element.\n\t *\n\t * @param {HTMLElement} element Element to pick up options\n\t * from, either a slide or an animation target\n\t * @param {Object} [inheritedOptions] Optional set of existing\n\t * options\n\t */\n\tgetAutoAnimateOptions( element, inheritedOptions ) {\n\n\t\tlet options = {\n\t\t\teasing: this.Reveal.getConfig().autoAnimateEasing,\n\t\t\tduration: this.Reveal.getConfig().autoAnimateDuration,\n\t\t\tdelay: 0\n\t\t};\n\n\t\toptions = extend( options, inheritedOptions );\n\n\t\t// Inherit options from parent elements\n\t\tif( element.parentNode ) {\n\t\t\tlet autoAnimatedParent = closest( element.parentNode, '[data-auto-animate-target]' );\n\t\t\tif( autoAnimatedParent ) {\n\t\t\t\toptions = this.getAutoAnimateOptions( autoAnimatedParent, options );\n\t\t\t}\n\t\t}\n\n\t\tif( element.dataset.autoAnimateEasing ) {\n\t\t\toptions.easing = element.dataset.autoAnimateEasing;\n\t\t}\n\n\t\tif( element.dataset.autoAnimateDuration ) {\n\t\t\toptions.duration = parseFloat( element.dataset.autoAnimateDuration );\n\t\t}\n\n\t\tif( element.dataset.autoAnimateDelay ) {\n\t\t\toptions.delay = parseFloat( element.dataset.autoAnimateDelay );\n\t\t}\n\n\t\treturn options;\n\n\t}\n\n\t/**\n\t * Returns an object containing all of the properties\n\t * that can be auto-animated for the given element and\n\t * their current computed values.\n\t *\n\t * @param {String} direction 'from' or 'to'\n\t */\n\tgetAutoAnimatableProperties( direction, element, elementOptions ) {\n\n\t\tlet config = this.Reveal.getConfig();\n\n\t\tlet properties = { styles: [] };\n\n\t\t// Position and size\n\t\tif( elementOptions.translate !== false || elementOptions.scale !== false ) {\n\t\t\tlet bounds;\n\n\t\t\t// Custom auto-animate may optionally return a custom tailored\n\t\t\t// measurement function\n\t\t\tif( typeof elementOptions.measure === 'function' ) {\n\t\t\t\tbounds = elementOptions.measure( element );\n\t\t\t}\n\t\t\telse {\n\t\t\t\tif( config.center ) {\n\t\t\t\t\t// More precise, but breaks when used in combination\n\t\t\t\t\t// with zoom for scaling the deck ¯\\_(ツ)_/¯\n\t\t\t\t\tbounds = element.getBoundingClientRect();\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tlet scale = this.Reveal.getScale();\n\t\t\t\t\tbounds = {\n\t\t\t\t\t\tx: element.offsetLeft * scale,\n\t\t\t\t\t\ty: element.offsetTop * scale,\n\t\t\t\t\t\twidth: element.offsetWidth * scale,\n\t\t\t\t\t\theight: element.offsetHeight * scale\n\t\t\t\t\t};\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tproperties.x = bounds.x;\n\t\t\tproperties.y = bounds.y;\n\t\t\tproperties.width = bounds.width;\n\t\t\tproperties.height = bounds.height;\n\t\t}\n\n\t\tconst computedStyles = getComputedStyle( element );\n\n\t\t// CSS styles\n\t\t( elementOptions.styles || config.autoAnimateStyles ).forEach( style => {\n\t\t\tlet value;\n\n\t\t\t// `style` is either the property name directly, or an object\n\t\t\t// definition of a style property\n\t\t\tif( typeof style === 'string' ) style = { property: style };\n\n\t\t\tif( typeof style.from !== 'undefined' && direction === 'from' ) {\n\t\t\t\tvalue = { value: style.from, explicitValue: true };\n\t\t\t}\n\t\t\telse if( typeof style.to !== 'undefined' && direction === 'to' ) {\n\t\t\t\tvalue = { value: style.to, explicitValue: true };\n\t\t\t}\n\t\t\telse {\n\t\t\t\tvalue = computedStyles[style.property];\n\t\t\t}\n\n\t\t\tif( value !== '' ) {\n\t\t\t\tproperties.styles[style.property] = value;\n\t\t\t}\n\t\t} );\n\n\t\treturn properties;\n\n\t}\n\n\t/**\n\t * Get a list of all element pairs that we can animate\n\t * between the given slides.\n\t *\n\t * @param {HTMLElement} fromSlide\n\t * @param {HTMLElement} toSlide\n\t *\n\t * @return {Array} Each value is an array where [0] is\n\t * the element we're animating from and [1] is the\n\t * element we're animating to\n\t */\n\tgetAutoAnimatableElements( fromSlide, toSlide ) {\n\n\t\tlet matcher = typeof this.Reveal.getConfig().autoAnimateMatcher === 'function' ? this.Reveal.getConfig().autoAnimateMatcher : this.getAutoAnimatePairs;\n\n\t\tlet pairs = matcher.call( this, fromSlide, toSlide );\n\n\t\tlet reserved = [];\n\n\t\t// Remove duplicate pairs\n\t\treturn pairs.filter( ( pair, index ) => {\n\t\t\tif( reserved.indexOf( pair.to ) === -1 ) {\n\t\t\t\treserved.push( pair.to );\n\t\t\t\treturn true;\n\t\t\t}\n\t\t} );\n\n\t}\n\n\t/**\n\t * Identifies matching elements between slides.\n\t *\n\t * You can specify a custom matcher function by using\n\t * the `autoAnimateMatcher` config option.\n\t */\n\tgetAutoAnimatePairs( fromSlide, toSlide ) {\n\n\t\tlet pairs = [];\n\n\t\tconst codeNodes = 'pre';\n\t\tconst textNodes = 'h1, h2, h3, h4, h5, h6, p, li';\n\t\tconst mediaNodes = 'img, video, iframe';\n\n\t\t// Eplicit matches via data-id\n\t\tthis.findAutoAnimateMatches( pairs, fromSlide, toSlide, '[data-id]', node => {\n\t\t\treturn node.nodeName + ':::' + node.getAttribute( 'data-id' );\n\t\t} );\n\n\t\t// Text\n\t\tthis.findAutoAnimateMatches( pairs, fromSlide, toSlide, textNodes, node => {\n\t\t\treturn node.nodeName + ':::' + node.innerText;\n\t\t} );\n\n\t\t// Media\n\t\tthis.findAutoAnimateMatches( pairs, fromSlide, toSlide, mediaNodes, node => {\n\t\t\treturn node.nodeName + ':::' + ( node.getAttribute( 'src' ) || node.getAttribute( 'data-src' ) );\n\t\t} );\n\n\t\t// Code\n\t\tthis.findAutoAnimateMatches( pairs, fromSlide, toSlide, codeNodes, node => {\n\t\t\treturn node.nodeName + ':::' + node.innerText;\n\t\t} );\n\n\t\tpairs.forEach( pair => {\n\n\t\t\t// Disable scale transformations on text nodes, we transition\n\t\t\t// each individual text property instead\n\t\t\tif( matches( pair.from, textNodes ) ) {\n\t\t\t\tpair.options = { scale: false };\n\t\t\t}\n\t\t\t// Animate individual lines of code\n\t\t\telse if( matches( pair.from, codeNodes ) ) {\n\n\t\t\t\t// Transition the code block's width and height instead of scaling\n\t\t\t\t// to prevent its content from being squished\n\t\t\t\tpair.options = { scale: false, styles: [ 'width', 'height' ] };\n\n\t\t\t\t// Lines of code\n\t\t\t\tthis.findAutoAnimateMatches( pairs, pair.from, pair.to, '.hljs .hljs-ln-code', node => {\n\t\t\t\t\treturn node.textContent;\n\t\t\t\t}, {\n\t\t\t\t\tscale: false,\n\t\t\t\t\tstyles: [],\n\t\t\t\t\tmeasure: this.getLocalBoundingBox.bind( this )\n\t\t\t\t} );\n\n\t\t\t\t// Line numbers\n\t\t\t\tthis.findAutoAnimateMatches( pairs, pair.from, pair.to, '.hljs .hljs-ln-line[data-line-number]', node => {\n\t\t\t\t\treturn node.getAttribute( 'data-line-number' );\n\t\t\t\t}, {\n\t\t\t\t\tscale: false,\n\t\t\t\t\tstyles: [ 'width' ],\n\t\t\t\t\tmeasure: this.getLocalBoundingBox.bind( this )\n\t\t\t\t} );\n\n\t\t\t}\n\n\t\t}, this );\n\n\t\treturn pairs;\n\n\t}\n\n\t/**\n\t * Helper method which returns a bounding box based on\n\t * the given elements offset coordinates.\n\t *\n\t * @param {HTMLElement} element\n\t * @return {Object} x, y, width, height\n\t */\n\tgetLocalBoundingBox( element ) {\n\n\t\tconst presentationScale = this.Reveal.getScale();\n\n\t\treturn {\n\t\t\tx: Math.round( ( element.offsetLeft * presentationScale ) * 100 ) / 100,\n\t\t\ty: Math.round( ( element.offsetTop * presentationScale ) * 100 ) / 100,\n\t\t\twidth: Math.round( ( element.offsetWidth * presentationScale ) * 100 ) / 100,\n\t\t\theight: Math.round( ( element.offsetHeight * presentationScale ) * 100 ) / 100\n\t\t};\n\n\t}\n\n\t/**\n\t * Finds matching elements between two slides.\n\t *\n\t * @param {Array} pairs \tList of pairs to push matches to\n\t * @param {HTMLElement} fromScope Scope within the from element exists\n\t * @param {HTMLElement} toScope Scope within the to element exists\n\t * @param {String} selector CSS selector of the element to match\n\t * @param {Function} serializer A function that accepts an element and returns\n\t * a stringified ID based on its contents\n\t * @param {Object} animationOptions Optional config options for this pair\n\t */\n\tfindAutoAnimateMatches( pairs, fromScope, toScope, selector, serializer, animationOptions ) {\n\n\t\tlet fromMatches = {};\n\t\tlet toMatches = {};\n\n\t\t[].slice.call( fromScope.querySelectorAll( selector ) ).forEach( ( element, i ) => {\n\t\t\tconst key = serializer( element );\n\t\t\tif( typeof key === 'string' && key.length ) {\n\t\t\t\tfromMatches[key] = fromMatches[key] || [];\n\t\t\t\tfromMatches[key].push( element );\n\t\t\t}\n\t\t} );\n\n\t\t[].slice.call( toScope.querySelectorAll( selector ) ).forEach( ( element, i ) => {\n\t\t\tconst key = serializer( element );\n\t\t\ttoMatches[key] = toMatches[key] || [];\n\t\t\ttoMatches[key].push( element );\n\n\t\t\tlet fromElement;\n\n\t\t\t// Retrieve the 'from' element\n\t\t\tif( fromMatches[key] ) {\n\t\t\t\tconst pimaryIndex = toMatches[key].length - 1;\n\t\t\t\tconst secondaryIndex = fromMatches[key].length - 1;\n\n\t\t\t\t// If there are multiple identical from elements, retrieve\n\t\t\t\t// the one at the same index as our to-element.\n\t\t\t\tif( fromMatches[key][ pimaryIndex ] ) {\n\t\t\t\t\tfromElement = fromMatches[key][ pimaryIndex ];\n\t\t\t\t\tfromMatches[key][ pimaryIndex ] = null;\n\t\t\t\t}\n\t\t\t\t// If there are no matching from-elements at the same index,\n\t\t\t\t// use the last one.\n\t\t\t\telse if( fromMatches[key][ secondaryIndex ] ) {\n\t\t\t\t\tfromElement = fromMatches[key][ secondaryIndex ];\n\t\t\t\t\tfromMatches[key][ secondaryIndex ] = null;\n\t\t\t\t}\n\t\t\t}\n\n\t\t\t// If we've got a matching pair, push it to the list of pairs\n\t\t\tif( fromElement ) {\n\t\t\t\tpairs.push({\n\t\t\t\t\tfrom: fromElement,\n\t\t\t\t\tto: element,\n\t\t\t\t\toptions: animationOptions\n\t\t\t\t});\n\t\t\t}\n\t\t} );\n\n\t}\n\n\t/**\n\t * Returns a all elements within the given scope that should\n\t * be considered unmatched in an auto-animate transition. If\n\t * fading of unmatched elements is turned on, these elements\n\t * will fade when going between auto-animate slides.\n\t *\n\t * Note that parents of auto-animate targets are NOT considerd\n\t * unmatched since fading them would break the auto-animation.\n\t *\n\t * @param {HTMLElement} rootElement\n\t * @return {Array}\n\t */\n\tgetUnmatchedAutoAnimateElements( rootElement ) {\n\n\t\treturn [].slice.call( rootElement.children ).reduce( ( result, element ) => {\n\n\t\t\tconst containsAnimatedElements = element.querySelector( '[data-auto-animate-target]' );\n\n\t\t\t// The element is unmatched if\n\t\t\t// - It is not an auto-animate target\n\t\t\t// - It does not contain any auto-animate targets\n\t\t\tif( !element.hasAttribute( 'data-auto-animate-target' ) && !containsAnimatedElements ) {\n\t\t\t\tresult.push( element );\n\t\t\t}\n\n\t\t\tif( element.querySelector( '[data-auto-animate-target]' ) ) {\n\t\t\t\tresult = result.concat( this.getUnmatchedAutoAnimateElements( element ) );\n\t\t\t}\n\n\t\t\treturn result;\n\n\t\t}, [] );\n\n\t}\n\n}\n","import { extend, queryAll } from '../utils/util.js'\n\n/**\n * Handles sorting and navigation of slide fragments.\n * Fragments are elements within a slide that are\n * revealed/animated incrementally.\n */\nexport default class Fragments {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t}\n\n\t/**\n\t * Called when the reveal.js config is updated.\n\t */\n\tconfigure( config, oldConfig ) {\n\n\t\tif( config.fragments === false ) {\n\t\t\tthis.disable();\n\t\t}\n\t\telse if( oldConfig.fragments === false ) {\n\t\t\tthis.enable();\n\t\t}\n\n\t}\n\n\t/**\n\t * If fragments are disabled in the deck, they should all be\n\t * visible rather than stepped through.\n\t */\n\tdisable() {\n\n\t\tqueryAll( this.Reveal.getSlidesElement(), '.fragment' ).forEach( element => {\n\t\t\telement.classList.add( 'visible' );\n\t\t\telement.classList.remove( 'current-fragment' );\n\t\t} );\n\n\t}\n\n\t/**\n\t * Reverse of #disable(). Only called if fragments have\n\t * previously been disabled.\n\t */\n\tenable() {\n\n\t\tqueryAll( this.Reveal.getSlidesElement(), '.fragment' ).forEach( element => {\n\t\t\telement.classList.remove( 'visible' );\n\t\t\telement.classList.remove( 'current-fragment' );\n\t\t} );\n\n\t}\n\n\t/**\n\t * Returns an object describing the available fragment\n\t * directions.\n\t *\n\t * @return {{prev: boolean, next: boolean}}\n\t */\n\tavailableRoutes() {\n\n\t\tlet currentSlide = this.Reveal.getCurrentSlide();\n\t\tif( currentSlide && this.Reveal.getConfig().fragments ) {\n\t\t\tlet fragments = currentSlide.querySelectorAll( '.fragment:not(.disabled)' );\n\t\t\tlet hiddenFragments = currentSlide.querySelectorAll( '.fragment:not(.disabled):not(.visible)' );\n\n\t\t\treturn {\n\t\t\t\tprev: fragments.length - hiddenFragments.length > 0,\n\t\t\t\tnext: !!hiddenFragments.length\n\t\t\t};\n\t\t}\n\t\telse {\n\t\t\treturn { prev: false, next: false };\n\t\t}\n\n\t}\n\n\t/**\n\t * Return a sorted fragments list, ordered by an increasing\n\t * \"data-fragment-index\" attribute.\n\t *\n\t * Fragments will be revealed in the order that they are returned by\n\t * this function, so you can use the index attributes to control the\n\t * order of fragment appearance.\n\t *\n\t * To maintain a sensible default fragment order, fragments are presumed\n\t * to be passed in document order. This function adds a \"fragment-index\"\n\t * attribute to each node if such an attribute is not already present,\n\t * and sets that attribute to an integer value which is the position of\n\t * the fragment within the fragments list.\n\t *\n\t * @param {object[]|*} fragments\n\t * @param {boolean} grouped If true the returned array will contain\n\t * nested arrays for all fragments with the same index\n\t * @return {object[]} sorted Sorted array of fragments\n\t */\n\tsort( fragments, grouped = false ) {\n\n\t\tfragments = Array.from( fragments );\n\n\t\tlet ordered = [],\n\t\t\tunordered = [],\n\t\t\tsorted = [];\n\n\t\t// Group ordered and unordered elements\n\t\tfragments.forEach( fragment => {\n\t\t\tif( fragment.hasAttribute( 'data-fragment-index' ) ) {\n\t\t\t\tlet index = parseInt( fragment.getAttribute( 'data-fragment-index' ), 10 );\n\n\t\t\t\tif( !ordered[index] ) {\n\t\t\t\t\tordered[index] = [];\n\t\t\t\t}\n\n\t\t\t\tordered[index].push( fragment );\n\t\t\t}\n\t\t\telse {\n\t\t\t\tunordered.push( [ fragment ] );\n\t\t\t}\n\t\t} );\n\n\t\t// Append fragments without explicit indices in their\n\t\t// DOM order\n\t\tordered = ordered.concat( unordered );\n\n\t\t// Manually count the index up per group to ensure there\n\t\t// are no gaps\n\t\tlet index = 0;\n\n\t\t// Push all fragments in their sorted order to an array,\n\t\t// this flattens the groups\n\t\tordered.forEach( group => {\n\t\t\tgroup.forEach( fragment => {\n\t\t\t\tsorted.push( fragment );\n\t\t\t\tfragment.setAttribute( 'data-fragment-index', index );\n\t\t\t} );\n\n\t\t\tindex ++;\n\t\t} );\n\n\t\treturn grouped === true ? ordered : sorted;\n\n\t}\n\n\t/**\n\t * Sorts and formats all of fragments in the\n\t * presentation.\n\t */\n\tsortAll() {\n\n\t\tthis.Reveal.getHorizontalSlides().forEach( horizontalSlide => {\n\n\t\t\tlet verticalSlides = queryAll( horizontalSlide, 'section' );\n\t\t\tverticalSlides.forEach( ( verticalSlide, y ) => {\n\n\t\t\t\tthis.sort( verticalSlide.querySelectorAll( '.fragment' ) );\n\n\t\t\t}, this );\n\n\t\t\tif( verticalSlides.length === 0 ) this.sort( horizontalSlide.querySelectorAll( '.fragment' ) );\n\n\t\t} );\n\n\t}\n\n\t/**\n\t * Refreshes the fragments on the current slide so that they\n\t * have the appropriate classes (.visible + .current-fragment).\n\t *\n\t * @param {number} [index] The index of the current fragment\n\t * @param {array} [fragments] Array containing all fragments\n\t * in the current slide\n\t *\n\t * @return {{shown: array, hidden: array}}\n\t */\n\tupdate( index, fragments ) {\n\n\t\tlet changedFragments = {\n\t\t\tshown: [],\n\t\t\thidden: []\n\t\t};\n\n\t\tlet currentSlide = this.Reveal.getCurrentSlide();\n\t\tif( currentSlide && this.Reveal.getConfig().fragments ) {\n\n\t\t\tfragments = fragments || this.sort( currentSlide.querySelectorAll( '.fragment' ) );\n\n\t\t\tif( fragments.length ) {\n\n\t\t\t\tlet maxIndex = 0;\n\n\t\t\t\tif( typeof index !== 'number' ) {\n\t\t\t\t\tlet currentFragment = this.sort( currentSlide.querySelectorAll( '.fragment.visible' ) ).pop();\n\t\t\t\t\tif( currentFragment ) {\n\t\t\t\t\t\tindex = parseInt( currentFragment.getAttribute( 'data-fragment-index' ) || 0, 10 );\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\tArray.from( fragments ).forEach( ( el, i ) => {\n\n\t\t\t\t\tif( el.hasAttribute( 'data-fragment-index' ) ) {\n\t\t\t\t\t\ti = parseInt( el.getAttribute( 'data-fragment-index' ), 10 );\n\t\t\t\t\t}\n\n\t\t\t\t\tmaxIndex = Math.max( maxIndex, i );\n\n\t\t\t\t\t// Visible fragments\n\t\t\t\t\tif( i <= index ) {\n\t\t\t\t\t\tlet wasVisible = el.classList.contains( 'visible' )\n\t\t\t\t\t\tel.classList.add( 'visible' );\n\t\t\t\t\t\tel.classList.remove( 'current-fragment' );\n\n\t\t\t\t\t\tif( i === index ) {\n\t\t\t\t\t\t\t// Announce the fragments one by one to the Screen Reader\n\t\t\t\t\t\t\tthis.Reveal.announceStatus( this.Reveal.getStatusText( el ) );\n\n\t\t\t\t\t\t\tel.classList.add( 'current-fragment' );\n\t\t\t\t\t\t\tthis.Reveal.slideContent.startEmbeddedContent( el );\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\tif( !wasVisible ) {\n\t\t\t\t\t\t\tchangedFragments.shown.push( el )\n\t\t\t\t\t\t\tthis.Reveal.dispatchEvent({\n\t\t\t\t\t\t\t\ttarget: el,\n\t\t\t\t\t\t\t\ttype: 'visible',\n\t\t\t\t\t\t\t\tbubbles: false\n\t\t\t\t\t\t\t});\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\t// Hidden fragments\n\t\t\t\t\telse {\n\t\t\t\t\t\tlet wasVisible = el.classList.contains( 'visible' )\n\t\t\t\t\t\tel.classList.remove( 'visible' );\n\t\t\t\t\t\tel.classList.remove( 'current-fragment' );\n\n\t\t\t\t\t\tif( wasVisible ) {\n\t\t\t\t\t\t\tthis.Reveal.slideContent.stopEmbeddedContent( el );\n\t\t\t\t\t\t\tchangedFragments.hidden.push( el );\n\t\t\t\t\t\t\tthis.Reveal.dispatchEvent({\n\t\t\t\t\t\t\t\ttarget: el,\n\t\t\t\t\t\t\t\ttype: 'hidden',\n\t\t\t\t\t\t\t\tbubbles: false\n\t\t\t\t\t\t\t});\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\n\t\t\t\t} );\n\n\t\t\t\t// Write the current fragment index to the slide .\n\t\t\t\t// This can be used by end users to apply styles based on\n\t\t\t\t// the current fragment index.\n\t\t\t\tindex = typeof index === 'number' ? index : -1;\n\t\t\t\tindex = Math.max( Math.min( index, maxIndex ), -1 );\n\t\t\t\tcurrentSlide.setAttribute( 'data-fragment', index );\n\n\t\t\t}\n\n\t\t}\n\n\t\treturn changedFragments;\n\n\t}\n\n\t/**\n\t * Formats the fragments on the given slide so that they have\n\t * valid indices. Call this if fragments are changed in the DOM\n\t * after reveal.js has already initialized.\n\t *\n\t * @param {HTMLElement} slide\n\t * @return {Array} a list of the HTML fragments that were synced\n\t */\n\tsync( slide = this.Reveal.getCurrentSlide() ) {\n\n\t\treturn this.sort( slide.querySelectorAll( '.fragment' ) );\n\n\t}\n\n\t/**\n\t * Navigate to the specified slide fragment.\n\t *\n\t * @param {?number} index The index of the fragment that\n\t * should be shown, -1 means all are invisible\n\t * @param {number} offset Integer offset to apply to the\n\t * fragment index\n\t *\n\t * @return {boolean} true if a change was made in any\n\t * fragments visibility as part of this call\n\t */\n\tgoto( index, offset = 0 ) {\n\n\t\tlet currentSlide = this.Reveal.getCurrentSlide();\n\t\tif( currentSlide && this.Reveal.getConfig().fragments ) {\n\n\t\t\tlet fragments = this.sort( currentSlide.querySelectorAll( '.fragment:not(.disabled)' ) );\n\t\t\tif( fragments.length ) {\n\n\t\t\t\t// If no index is specified, find the current\n\t\t\t\tif( typeof index !== 'number' ) {\n\t\t\t\t\tlet lastVisibleFragment = this.sort( currentSlide.querySelectorAll( '.fragment:not(.disabled).visible' ) ).pop();\n\n\t\t\t\t\tif( lastVisibleFragment ) {\n\t\t\t\t\t\tindex = parseInt( lastVisibleFragment.getAttribute( 'data-fragment-index' ) || 0, 10 );\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tindex = -1;\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\t// Apply the offset if there is one\n\t\t\t\tindex += offset;\n\n\t\t\t\tlet changedFragments = this.update( index, fragments );\n\n\t\t\t\tif( changedFragments.hidden.length ) {\n\t\t\t\t\tthis.Reveal.dispatchEvent({\n\t\t\t\t\t\ttype: 'fragmenthidden',\n\t\t\t\t\t\tdata: {\n\t\t\t\t\t\t\tfragment: changedFragments.hidden[0],\n\t\t\t\t\t\t\tfragments: changedFragments.hidden\n\t\t\t\t\t\t}\n\t\t\t\t\t});\n\t\t\t\t}\n\n\t\t\t\tif( changedFragments.shown.length ) {\n\t\t\t\t\tthis.Reveal.dispatchEvent({\n\t\t\t\t\t\ttype: 'fragmentshown',\n\t\t\t\t\t\tdata: {\n\t\t\t\t\t\t\tfragment: changedFragments.shown[0],\n\t\t\t\t\t\t\tfragments: changedFragments.shown\n\t\t\t\t\t\t}\n\t\t\t\t\t});\n\t\t\t\t}\n\n\t\t\t\tthis.Reveal.controls.update();\n\t\t\t\tthis.Reveal.progress.update();\n\n\t\t\t\tif( this.Reveal.getConfig().fragmentInURL ) {\n\t\t\t\t\tthis.Reveal.location.writeURL();\n\t\t\t\t}\n\n\t\t\t\treturn !!( changedFragments.shown.length || changedFragments.hidden.length );\n\n\t\t\t}\n\n\t\t}\n\n\t\treturn false;\n\n\t}\n\n\t/**\n\t * Navigate to the next slide fragment.\n\t *\n\t * @return {boolean} true if there was a next fragment,\n\t * false otherwise\n\t */\n\tnext() {\n\n\t\treturn this.goto( null, 1 );\n\n\t}\n\n\t/**\n\t * Navigate to the previous slide fragment.\n\t *\n\t * @return {boolean} true if there was a previous fragment,\n\t * false otherwise\n\t */\n\tprev() {\n\n\t\treturn this.goto( null, -1 );\n\n\t}\n\n}","import { SLIDES_SELECTOR } from '../utils/constants.js'\nimport { extend, queryAll, transformElement } from '../utils/util.js'\n\n/**\n * Handles all logic related to the overview mode\n * (birds-eye view of all slides).\n */\nexport default class Overview {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t\tthis.active = false;\n\n\t\tthis.onSlideClicked = this.onSlideClicked.bind( this );\n\n\t}\n\n\t/**\n\t * Displays the overview of slides (quick nav) by scaling\n\t * down and arranging all slide elements.\n\t */\n\tactivate() {\n\n\t\t// Only proceed if enabled in config\n\t\tif( this.Reveal.getConfig().overview && !this.isActive() ) {\n\n\t\t\tthis.active = true;\n\n\t\t\tthis.Reveal.getRevealElement().classList.add( 'overview' );\n\n\t\t\t// Don't auto-slide while in overview mode\n\t\t\tthis.Reveal.cancelAutoSlide();\n\n\t\t\t// Move the backgrounds element into the slide container to\n\t\t\t// that the same scaling is applied\n\t\t\tthis.Reveal.getSlidesElement().appendChild( this.Reveal.getBackgroundsElement() );\n\n\t\t\t// Clicking on an overview slide navigates to it\n\t\t\tqueryAll( this.Reveal.getRevealElement(), SLIDES_SELECTOR ).forEach( slide => {\n\t\t\t\tif( !slide.classList.contains( 'stack' ) ) {\n\t\t\t\t\tslide.addEventListener( 'click', this.onSlideClicked, true );\n\t\t\t\t}\n\t\t\t} );\n\n\t\t\t// Calculate slide sizes\n\t\t\tconst margin = 70;\n\t\t\tconst slideSize = this.Reveal.getComputedSlideSize();\n\t\t\tthis.overviewSlideWidth = slideSize.width + margin;\n\t\t\tthis.overviewSlideHeight = slideSize.height + margin;\n\n\t\t\t// Reverse in RTL mode\n\t\t\tif( this.Reveal.getConfig().rtl ) {\n\t\t\t\tthis.overviewSlideWidth = -this.overviewSlideWidth;\n\t\t\t}\n\n\t\t\tthis.Reveal.updateSlidesVisibility();\n\n\t\t\tthis.layout();\n\t\t\tthis.update();\n\n\t\t\tthis.Reveal.layout();\n\n\t\t\tconst indices = this.Reveal.getIndices();\n\n\t\t\t// Notify observers of the overview showing\n\t\t\tthis.Reveal.dispatchEvent({\n\t\t\t\ttype: 'overviewshown',\n\t\t\t\tdata: {\n\t\t\t\t\t'indexh': indices.h,\n\t\t\t\t\t'indexv': indices.v,\n\t\t\t\t\t'currentSlide': this.Reveal.getCurrentSlide()\n\t\t\t\t}\n\t\t\t});\n\n\t\t}\n\n\t}\n\n\t/**\n\t * Uses CSS transforms to position all slides in a grid for\n\t * display inside of the overview mode.\n\t */\n\tlayout() {\n\n\t\t// Layout slides\n\t\tthis.Reveal.getHorizontalSlides().forEach( ( hslide, h ) => {\n\t\t\thslide.setAttribute( 'data-index-h', h );\n\t\t\ttransformElement( hslide, 'translate3d(' + ( h * this.overviewSlideWidth ) + 'px, 0, 0)' );\n\n\t\t\tif( hslide.classList.contains( 'stack' ) ) {\n\n\t\t\t\tqueryAll( hslide, 'section' ).forEach( ( vslide, v ) => {\n\t\t\t\t\tvslide.setAttribute( 'data-index-h', h );\n\t\t\t\t\tvslide.setAttribute( 'data-index-v', v );\n\n\t\t\t\t\ttransformElement( vslide, 'translate3d(0, ' + ( v * this.overviewSlideHeight ) + 'px, 0)' );\n\t\t\t\t} );\n\n\t\t\t}\n\t\t} );\n\n\t\t// Layout slide backgrounds\n\t\tArray.from( this.Reveal.getBackgroundsElement().childNodes ).forEach( ( hbackground, h ) => {\n\t\t\ttransformElement( hbackground, 'translate3d(' + ( h * this.overviewSlideWidth ) + 'px, 0, 0)' );\n\n\t\t\tqueryAll( hbackground, '.slide-background' ).forEach( ( vbackground, v ) => {\n\t\t\t\ttransformElement( vbackground, 'translate3d(0, ' + ( v * this.overviewSlideHeight ) + 'px, 0)' );\n\t\t\t} );\n\t\t} );\n\n\t}\n\n\t/**\n\t * Moves the overview viewport to the current slides.\n\t * Called each time the current slide changes.\n\t */\n\tupdate() {\n\n\t\tconst vmin = Math.min( window.innerWidth, window.innerHeight );\n\t\tconst scale = Math.max( vmin / 5, 150 ) / vmin;\n\t\tconst indices = this.Reveal.getIndices();\n\n\t\tthis.Reveal.transformSlides( {\n\t\t\toverview: [\n\t\t\t\t'scale('+ scale +')',\n\t\t\t\t'translateX('+ ( -indices.h * this.overviewSlideWidth ) +'px)',\n\t\t\t\t'translateY('+ ( -indices.v * this.overviewSlideHeight ) +'px)'\n\t\t\t].join( ' ' )\n\t\t} );\n\n\t}\n\n\t/**\n\t * Exits the slide overview and enters the currently\n\t * active slide.\n\t */\n\tdeactivate() {\n\n\t\t// Only proceed if enabled in config\n\t\tif( this.Reveal.getConfig().overview ) {\n\n\t\t\tthis.active = false;\n\n\t\t\tthis.Reveal.getRevealElement().classList.remove( 'overview' );\n\n\t\t\t// Temporarily add a class so that transitions can do different things\n\t\t\t// depending on whether they are exiting/entering overview, or just\n\t\t\t// moving from slide to slide\n\t\t\tthis.Reveal.getRevealElement().classList.add( 'overview-deactivating' );\n\n\t\t\tsetTimeout( () => {\n\t\t\t\tthis.Reveal.getRevealElement().classList.remove( 'overview-deactivating' );\n\t\t\t}, 1 );\n\n\t\t\t// Move the background element back out\n\t\t\tthis.Reveal.getRevealElement().appendChild( this.Reveal.getBackgroundsElement() );\n\n\t\t\t// Clean up changes made to slides\n\t\t\tqueryAll( this.Reveal.getRevealElement(), SLIDES_SELECTOR ).forEach( slide => {\n\t\t\t\ttransformElement( slide, '' );\n\n\t\t\t\tslide.removeEventListener( 'click', this.onSlideClicked, true );\n\t\t\t} );\n\n\t\t\t// Clean up changes made to backgrounds\n\t\t\tqueryAll( this.Reveal.getBackgroundsElement(), '.slide-background' ).forEach( background => {\n\t\t\t\ttransformElement( background, '' );\n\t\t\t} );\n\n\t\t\tthis.Reveal.transformSlides( { overview: '' } );\n\n\t\t\tconst indices = this.Reveal.getIndices();\n\n\t\t\tthis.Reveal.slide( indices.h, indices.v );\n\t\t\tthis.Reveal.layout();\n\t\t\tthis.Reveal.cueAutoSlide();\n\n\t\t\t// Notify observers of the overview hiding\n\t\t\tthis.Reveal.dispatchEvent({\n\t\t\t\ttype: 'overviewhidden',\n\t\t\t\tdata: {\n\t\t\t\t\t'indexh': indices.h,\n\t\t\t\t\t'indexv': indices.v,\n\t\t\t\t\t'currentSlide': this.Reveal.getCurrentSlide()\n\t\t\t\t}\n\t\t\t});\n\n\t\t}\n\t}\n\n\t/**\n\t * Toggles the slide overview mode on and off.\n\t *\n\t * @param {Boolean} [override] Flag which overrides the\n\t * toggle logic and forcibly sets the desired state. True means\n\t * overview is open, false means it's closed.\n\t */\n\ttoggle( override ) {\n\n\t\tif( typeof override === 'boolean' ) {\n\t\t\toverride ? this.activate() : this.deactivate();\n\t\t}\n\t\telse {\n\t\t\tthis.isActive() ? this.deactivate() : this.activate();\n\t\t}\n\n\t}\n\n\t/**\n\t * Checks if the overview is currently active.\n\t *\n\t * @return {Boolean} true if the overview is active,\n\t * false otherwise\n\t */\n\tisActive() {\n\n\t\treturn this.active;\n\n\t}\n\n\t/**\n\t * Invoked when a slide is and we're in the overview.\n\t *\n\t * @param {object} event\n\t */\n\tonSlideClicked( event ) {\n\n\t\tif( this.isActive() ) {\n\t\t\tevent.preventDefault();\n\n\t\t\tlet element = event.target;\n\n\t\t\twhile( element && !element.nodeName.match( /section/gi ) ) {\n\t\t\t\telement = element.parentNode;\n\t\t\t}\n\n\t\t\tif( element && !element.classList.contains( 'disabled' ) ) {\n\n\t\t\t\tthis.deactivate();\n\n\t\t\t\tif( element.nodeName.match( /section/gi ) ) {\n\t\t\t\t\tlet h = parseInt( element.getAttribute( 'data-index-h' ), 10 ),\n\t\t\t\t\t\tv = parseInt( element.getAttribute( 'data-index-v' ), 10 );\n\n\t\t\t\t\tthis.Reveal.slide( h, v );\n\t\t\t\t}\n\n\t\t\t}\n\t\t}\n\n\t}\n\n}","import { enterFullscreen } from '../utils/util.js'\n\n/**\n * Handles all reveal.js keyboard interactions.\n */\nexport default class Keyboard {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t\t// A key:value map of keyboard keys and descriptions of\n\t\t// the actions they trigger\n\t\tthis.shortcuts = {};\n\n\t\t// Holds custom key code mappings\n\t\tthis.bindings = {};\n\n\t\tthis.onDocumentKeyDown = this.onDocumentKeyDown.bind( this );\n\t\tthis.onDocumentKeyPress = this.onDocumentKeyPress.bind( this );\n\n\t}\n\n\t/**\n\t * Called when the reveal.js config is updated.\n\t */\n\tconfigure( config, oldConfig ) {\n\n\t\tif( config.navigationMode === 'linear' ) {\n\t\t\tthis.shortcuts['→ , ↓ , SPACE , N , L , J'] = 'Next slide';\n\t\t\tthis.shortcuts['← , ↑ , P , H , K'] = 'Previous slide';\n\t\t}\n\t\telse {\n\t\t\tthis.shortcuts['N , SPACE'] = 'Next slide';\n\t\t\tthis.shortcuts['P , Shift SPACE'] = 'Previous slide';\n\t\t\tthis.shortcuts['← , H'] = 'Navigate left';\n\t\t\tthis.shortcuts['→ , L'] = 'Navigate right';\n\t\t\tthis.shortcuts['↑ , K'] = 'Navigate up';\n\t\t\tthis.shortcuts['↓ , J'] = 'Navigate down';\n\t\t}\n\n\t\tthis.shortcuts['Alt + ←/↑/→/↓'] = 'Navigate without fragments';\n\t\tthis.shortcuts['Shift + ←/↑/→/↓'] = 'Jump to first/last slide';\n\t\tthis.shortcuts['B , .'] = 'Pause';\n\t\tthis.shortcuts['F'] = 'Fullscreen';\n\t\tthis.shortcuts['ESC, O'] = 'Slide overview';\n\n\t}\n\n\t/**\n\t * Starts listening for keyboard events.\n\t */\n\tbind() {\n\n\t\tdocument.addEventListener( 'keydown', this.onDocumentKeyDown, false );\n\t\tdocument.addEventListener( 'keypress', this.onDocumentKeyPress, false );\n\n\t}\n\n\t/**\n\t * Stops listening for keyboard events.\n\t */\n\tunbind() {\n\n\t\tdocument.removeEventListener( 'keydown', this.onDocumentKeyDown, false );\n\t\tdocument.removeEventListener( 'keypress', this.onDocumentKeyPress, false );\n\n\t}\n\n\t/**\n\t * Add a custom key binding with optional description to\n\t * be added to the help screen.\n\t */\n\taddKeyBinding( binding, callback ) {\n\n\t\tif( typeof binding === 'object' && binding.keyCode ) {\n\t\t\tthis.bindings[binding.keyCode] = {\n\t\t\t\tcallback: callback,\n\t\t\t\tkey: binding.key,\n\t\t\t\tdescription: binding.description\n\t\t\t};\n\t\t}\n\t\telse {\n\t\t\tthis.bindings[binding] = {\n\t\t\t\tcallback: callback,\n\t\t\t\tkey: null,\n\t\t\t\tdescription: null\n\t\t\t};\n\t\t}\n\n\t}\n\n\t/**\n\t * Removes the specified custom key binding.\n\t */\n\tremoveKeyBinding( keyCode ) {\n\n\t\tdelete this.bindings[keyCode];\n\n\t}\n\n\t/**\n\t * Programmatically triggers a keyboard event\n\t *\n\t * @param {int} keyCode\n\t */\n\ttriggerKey( keyCode ) {\n\n\t\tthis.onDocumentKeyDown( { keyCode } );\n\n\t}\n\n\t/**\n\t * Registers a new shortcut to include in the help overlay\n\t *\n\t * @param {String} key\n\t * @param {String} value\n\t */\n\tregisterKeyboardShortcut( key, value ) {\n\n\t\tthis.shortcuts[key] = value;\n\n\t}\n\n\tgetShortcuts() {\n\n\t\treturn this.shortcuts;\n\n\t}\n\n\tgetBindings() {\n\n\t\treturn this.bindings;\n\n\t}\n\n\t/**\n\t * Handler for the document level 'keypress' event.\n\t *\n\t * @param {object} event\n\t */\n\tonDocumentKeyPress( event ) {\n\n\t\t// Check if the pressed key is question mark\n\t\tif( event.shiftKey && event.charCode === 63 ) {\n\t\t\tthis.Reveal.toggleHelp();\n\t\t}\n\n\t}\n\n\t/**\n\t * Handler for the document level 'keydown' event.\n\t *\n\t * @param {object} event\n\t */\n\tonDocumentKeyDown( event ) {\n\n\t\tlet config = this.Reveal.getConfig();\n\n\t\t// If there's a condition specified and it returns false,\n\t\t// ignore this event\n\t\tif( typeof config.keyboardCondition === 'function' && config.keyboardCondition(event) === false ) {\n\t\t\treturn true;\n\t\t}\n\n\t\t// If keyboardCondition is set, only capture keyboard events\n\t\t// for embedded decks when they are focused\n\t\tif( config.keyboardCondition === 'focused' && !this.Reveal.isFocused() ) {\n\t\t\treturn true;\n\t\t}\n\n\t\t// Shorthand\n\t\tlet keyCode = event.keyCode;\n\n\t\t// Remember if auto-sliding was paused so we can toggle it\n\t\tlet autoSlideWasPaused = !this.Reveal.isAutoSliding();\n\n\t\tthis.Reveal.onUserInput( event );\n\n\t\t// Is there a focused element that could be using the keyboard?\n\t\tlet activeElementIsCE = document.activeElement && document.activeElement.isContentEditable === true;\n\t\tlet activeElementIsInput = document.activeElement && document.activeElement.tagName && /input|textarea/i.test( document.activeElement.tagName );\n\t\tlet activeElementIsNotes = document.activeElement && document.activeElement.className && /speaker-notes/i.test( document.activeElement.className);\n\n\t\t// Whitelist certain modifiers for slide navigation shortcuts\n\t\tlet isNavigationKey = [32, 37, 38, 39, 40, 78, 80].indexOf( event.keyCode ) !== -1;\n\n\t\t// Prevent all other events when a modifier is pressed\n\t\tlet unusedModifier = \t!( isNavigationKey && event.shiftKey || event.altKey ) &&\n\t\t\t\t\t\t\t\t( event.shiftKey || event.altKey || event.ctrlKey || event.metaKey );\n\n\t\t// Disregard the event if there's a focused element or a\n\t\t// keyboard modifier key is present\n\t\tif( activeElementIsCE || activeElementIsInput || activeElementIsNotes || unusedModifier ) return;\n\n\t\t// While paused only allow resume keyboard events; 'b', 'v', '.'\n\t\tlet resumeKeyCodes = [66,86,190,191];\n\t\tlet key;\n\n\t\t// Custom key bindings for togglePause should be able to resume\n\t\tif( typeof config.keyboard === 'object' ) {\n\t\t\tfor( key in config.keyboard ) {\n\t\t\t\tif( config.keyboard[key] === 'togglePause' ) {\n\t\t\t\t\tresumeKeyCodes.push( parseInt( key, 10 ) );\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\tif( this.Reveal.isPaused() && resumeKeyCodes.indexOf( keyCode ) === -1 ) {\n\t\t\treturn false;\n\t\t}\n\n\t\t// Use linear navigation if we're configured to OR if\n\t\t// the presentation is one-dimensional\n\t\tlet useLinearMode = config.navigationMode === 'linear' || !this.Reveal.hasHorizontalSlides() || !this.Reveal.hasVerticalSlides();\n\n\t\tlet triggered = false;\n\n\t\t// 1. User defined key bindings\n\t\tif( typeof config.keyboard === 'object' ) {\n\n\t\t\tfor( key in config.keyboard ) {\n\n\t\t\t\t// Check if this binding matches the pressed key\n\t\t\t\tif( parseInt( key, 10 ) === keyCode ) {\n\n\t\t\t\t\tlet value = config.keyboard[ key ];\n\n\t\t\t\t\t// Callback function\n\t\t\t\t\tif( typeof value === 'function' ) {\n\t\t\t\t\t\tvalue.apply( null, [ event ] );\n\t\t\t\t\t}\n\t\t\t\t\t// String shortcuts to reveal.js API\n\t\t\t\t\telse if( typeof value === 'string' && typeof this.Reveal[ value ] === 'function' ) {\n\t\t\t\t\t\tthis.Reveal[ value ].call();\n\t\t\t\t\t}\n\n\t\t\t\t\ttriggered = true;\n\n\t\t\t\t}\n\n\t\t\t}\n\n\t\t}\n\n\t\t// 2. Registered custom key bindings\n\t\tif( triggered === false ) {\n\n\t\t\tfor( key in this.bindings ) {\n\n\t\t\t\t// Check if this binding matches the pressed key\n\t\t\t\tif( parseInt( key, 10 ) === keyCode ) {\n\n\t\t\t\t\tlet action = this.bindings[ key ].callback;\n\n\t\t\t\t\t// Callback function\n\t\t\t\t\tif( typeof action === 'function' ) {\n\t\t\t\t\t\taction.apply( null, [ event ] );\n\t\t\t\t\t}\n\t\t\t\t\t// String shortcuts to reveal.js API\n\t\t\t\t\telse if( typeof action === 'string' && typeof this.Reveal[ action ] === 'function' ) {\n\t\t\t\t\t\tthis.Reveal[ action ].call();\n\t\t\t\t\t}\n\n\t\t\t\t\ttriggered = true;\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\t// 3. System defined key bindings\n\t\tif( triggered === false ) {\n\n\t\t\t// Assume true and try to prove false\n\t\t\ttriggered = true;\n\n\t\t\t// P, PAGE UP\n\t\t\tif( keyCode === 80 || keyCode === 33 ) {\n\t\t\t\tthis.Reveal.prev({skipFragments: event.altKey});\n\t\t\t}\n\t\t\t// N, PAGE DOWN\n\t\t\telse if( keyCode === 78 || keyCode === 34 ) {\n\t\t\t\tthis.Reveal.next({skipFragments: event.altKey});\n\t\t\t}\n\t\t\t// H, LEFT\n\t\t\telse if( keyCode === 72 || keyCode === 37 ) {\n\t\t\t\tif( event.shiftKey ) {\n\t\t\t\t\tthis.Reveal.slide( 0 );\n\t\t\t\t}\n\t\t\t\telse if( !this.Reveal.overview.isActive() && useLinearMode ) {\n\t\t\t\t\tthis.Reveal.prev({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tthis.Reveal.left({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t}\n\t\t\t// L, RIGHT\n\t\t\telse if( keyCode === 76 || keyCode === 39 ) {\n\t\t\t\tif( event.shiftKey ) {\n\t\t\t\t\tthis.Reveal.slide( this.Reveal.getHorizontalSlides().length - 1 );\n\t\t\t\t}\n\t\t\t\telse if( !this.Reveal.overview.isActive() && useLinearMode ) {\n\t\t\t\t\tthis.Reveal.next({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tthis.Reveal.right({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t}\n\t\t\t// K, UP\n\t\t\telse if( keyCode === 75 || keyCode === 38 ) {\n\t\t\t\tif( event.shiftKey ) {\n\t\t\t\t\tthis.Reveal.slide( undefined, 0 );\n\t\t\t\t}\n\t\t\t\telse if( !this.Reveal.overview.isActive() && useLinearMode ) {\n\t\t\t\t\tthis.Reveal.prev({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tthis.Reveal.up({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t}\n\t\t\t// J, DOWN\n\t\t\telse if( keyCode === 74 || keyCode === 40 ) {\n\t\t\t\tif( event.shiftKey ) {\n\t\t\t\t\tthis.Reveal.slide( undefined, Number.MAX_VALUE );\n\t\t\t\t}\n\t\t\t\telse if( !this.Reveal.overview.isActive() && useLinearMode ) {\n\t\t\t\t\tthis.Reveal.next({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tthis.Reveal.down({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t}\n\t\t\t// HOME\n\t\t\telse if( keyCode === 36 ) {\n\t\t\t\tthis.Reveal.slide( 0 );\n\t\t\t}\n\t\t\t// END\n\t\t\telse if( keyCode === 35 ) {\n\t\t\t\tthis.Reveal.slide( this.Reveal.getHorizontalSlides().length - 1 );\n\t\t\t}\n\t\t\t// SPACE\n\t\t\telse if( keyCode === 32 ) {\n\t\t\t\tif( this.Reveal.overview.isActive() ) {\n\t\t\t\t\tthis.Reveal.overview.deactivate();\n\t\t\t\t}\n\t\t\t\tif( event.shiftKey ) {\n\t\t\t\t\tthis.Reveal.prev({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tthis.Reveal.next({skipFragments: event.altKey});\n\t\t\t\t}\n\t\t\t}\n\t\t\t// TWO-SPOT, SEMICOLON, B, V, PERIOD, LOGITECH PRESENTER TOOLS \"BLACK SCREEN\" BUTTON\n\t\t\telse if( keyCode === 58 || keyCode === 59 || keyCode === 66 || keyCode === 86 || keyCode === 190 || keyCode === 191 ) {\n\t\t\t\tthis.Reveal.togglePause();\n\t\t\t}\n\t\t\t// F\n\t\t\telse if( keyCode === 70 ) {\n\t\t\t\tenterFullscreen( config.embedded ? this.Reveal.getViewportElement() : document.documentElement );\n\t\t\t}\n\t\t\t// A\n\t\t\telse if( keyCode === 65 ) {\n\t\t\t\tif ( config.autoSlideStoppable ) {\n\t\t\t\t\tthis.Reveal.toggleAutoSlide( autoSlideWasPaused );\n\t\t\t\t}\n\t\t\t}\n\t\t\telse {\n\t\t\t\ttriggered = false;\n\t\t\t}\n\n\t\t}\n\n\t\t// If the input resulted in a triggered action we should prevent\n\t\t// the browsers default behavior\n\t\tif( triggered ) {\n\t\t\tevent.preventDefault && event.preventDefault();\n\t\t}\n\t\t// ESC or O key\n\t\telse if( keyCode === 27 || keyCode === 79 ) {\n\t\t\tif( this.Reveal.closeOverlay() === false ) {\n\t\t\t\tthis.Reveal.overview.toggle();\n\t\t\t}\n\n\t\t\tevent.preventDefault && event.preventDefault();\n\t\t}\n\n\t\t// If auto-sliding is enabled we need to cue up\n\t\t// another timeout\n\t\tthis.Reveal.cueAutoSlide();\n\n\t}\n\n}","/**\n * Reads and writes the URL based on reveal.js' current state.\n */\nexport default class Location {\n\n\t// The minimum number of milliseconds that must pass between\n\t// calls to history.replaceState\n\tMAX_REPLACE_STATE_FREQUENCY = 1000\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t\t// Delays updates to the URL due to a Chrome thumbnailer bug\n\t\tthis.writeURLTimeout = 0;\n\n\t\tthis.replaceStateTimestamp = 0;\n\n\t\tthis.onWindowHashChange = this.onWindowHashChange.bind( this );\n\n\t}\n\n\tbind() {\n\n\t\twindow.addEventListener( 'hashchange', this.onWindowHashChange, false );\n\n\t}\n\n\tunbind() {\n\n\t\twindow.removeEventListener( 'hashchange', this.onWindowHashChange, false );\n\n\t}\n\n\t/**\n\t * Returns the slide indices for the given hash link.\n\t *\n\t * @param {string} [hash] the hash string that we want to\n\t * find the indices for\n\t *\n\t * @returns slide indices or null\n\t */\n\tgetIndicesFromHash( hash=window.location.hash ) {\n\n\t\t// Attempt to parse the hash as either an index or name\n\t\tlet name = hash.replace( /^#\\/?/, '' );\n\t\tlet bits = name.split( '/' );\n\n\t\t// If the first bit is not fully numeric and there is a name we\n\t\t// can assume that this is a named link\n\t\tif( !/^[0-9]*$/.test( bits[0] ) && name.length ) {\n\t\t\tlet element;\n\n\t\t\tlet f;\n\n\t\t\t// Parse named links with fragments (#/named-link/2)\n\t\t\tif( /\\/[-\\d]+$/g.test( name ) ) {\n\t\t\t\tf = parseInt( name.split( '/' ).pop(), 10 );\n\t\t\t\tf = isNaN(f) ? undefined : f;\n\t\t\t\tname = name.split( '/' ).shift();\n\t\t\t}\n\n\t\t\t// Ensure the named link is a valid HTML ID attribute\n\t\t\ttry {\n\t\t\t\telement = document.getElementById( decodeURIComponent( name ) );\n\t\t\t}\n\t\t\tcatch ( error ) { }\n\n\t\t\tif( element ) {\n\t\t\t\treturn { ...this.Reveal.getIndices( element ), f };\n\t\t\t}\n\t\t}\n\t\telse {\n\t\t\tconst config = this.Reveal.getConfig();\n\t\t\tlet hashIndexBase = config.hashOneBasedIndex ? 1 : 0;\n\n\t\t\t// Read the index components of the hash\n\t\t\tlet h = ( parseInt( bits[0], 10 ) - hashIndexBase ) || 0,\n\t\t\t\tv = ( parseInt( bits[1], 10 ) - hashIndexBase ) || 0,\n\t\t\t\tf;\n\n\t\t\tif( config.fragmentInURL ) {\n\t\t\t\tf = parseInt( bits[2], 10 );\n\t\t\t\tif( isNaN( f ) ) {\n\t\t\t\t\tf = undefined;\n\t\t\t\t}\n\t\t\t}\n\n\t\t\treturn { h, v, f };\n\t\t}\n\n\t\t// The hash couldn't be parsed or no matching named link was found\n\t\treturn null\n\n\t}\n\n\t/**\n\t * Reads the current URL (hash) and navigates accordingly.\n\t */\n\treadURL() {\n\n\t\tconst currentIndices = this.Reveal.getIndices();\n\t\tconst newIndices = this.getIndicesFromHash();\n\n\t\tif( newIndices ) {\n\t\t\tif( ( newIndices.h !== currentIndices.h || newIndices.v !== currentIndices.v || newIndices.f !== undefined ) ) {\n\t\t\t\t\tthis.Reveal.slide( newIndices.h, newIndices.v, newIndices.f );\n\t\t\t}\n\t\t}\n\t\t// If no new indices are available, we're trying to navigate to\n\t\t// a slide hash that does not exist\n\t\telse {\n\t\t\tthis.Reveal.slide( currentIndices.h || 0, currentIndices.v || 0 );\n\t\t}\n\n\t}\n\n\t/**\n\t * Updates the page URL (hash) to reflect the current\n\t * state.\n\t *\n\t * @param {number} delay The time in ms to wait before\n\t * writing the hash\n\t */\n\twriteURL( delay ) {\n\n\t\tlet config = this.Reveal.getConfig();\n\t\tlet currentSlide = this.Reveal.getCurrentSlide();\n\n\t\t// Make sure there's never more than one timeout running\n\t\tclearTimeout( this.writeURLTimeout );\n\n\t\t// If a delay is specified, timeout this call\n\t\tif( typeof delay === 'number' ) {\n\t\t\tthis.writeURLTimeout = setTimeout( this.writeURL, delay );\n\t\t}\n\t\telse if( currentSlide ) {\n\n\t\t\tlet hash = this.getHash();\n\n\t\t\t// If we're configured to push to history OR the history\n\t\t\t// API is not avaialble.\n\t\t\tif( config.history ) {\n\t\t\t\twindow.location.hash = hash;\n\t\t\t}\n\t\t\t// If we're configured to reflect the current slide in the\n\t\t\t// URL without pushing to history.\n\t\t\telse if( config.hash ) {\n\t\t\t\t// If the hash is empty, don't add it to the URL\n\t\t\t\tif( hash === '/' ) {\n\t\t\t\t\tthis.debouncedReplaceState( window.location.pathname + window.location.search );\n\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tthis.debouncedReplaceState( '#' + hash );\n\t\t\t\t}\n\t\t\t}\n\t\t\t// UPDATE: The below nuking of all hash changes breaks\n\t\t\t// anchors on pages where reveal.js is running. Removed\n\t\t\t// in 4.0. Why was it here in the first place? ¯\\_(ツ)_/¯\n\t\t\t//\n\t\t\t// If history and hash are both disabled, a hash may still\n\t\t\t// be added to the URL by clicking on a href with a hash\n\t\t\t// target. Counter this by always removing the hash.\n\t\t\t// else {\n\t\t\t// \twindow.history.replaceState( null, null, window.location.pathname + window.location.search );\n\t\t\t// }\n\n\t\t}\n\n\t}\n\n\treplaceState( url ) {\n\n\t\twindow.history.replaceState( null, null, url );\n\t\tthis.replaceStateTimestamp = Date.now();\n\n\t}\n\n\tdebouncedReplaceState( url ) {\n\n\t\tclearTimeout( this.replaceStateTimeout );\n\n\t\tif( Date.now() - this.replaceStateTimestamp > this.MAX_REPLACE_STATE_FREQUENCY ) {\n\t\t\tthis.replaceState( url );\n\t\t}\n\t\telse {\n\t\t\tthis.replaceStateTimeout = setTimeout( () => this.replaceState( url ), this.MAX_REPLACE_STATE_FREQUENCY );\n\t\t}\n\n\t}\n\n\t/**\n\t * Return a hash URL that will resolve to the given slide location.\n\t *\n\t * @param {HTMLElement} [slide=currentSlide] The slide to link to\n\t */\n\tgetHash( slide ) {\n\n\t\tlet url = '/';\n\n\t\t// Attempt to create a named link based on the slide's ID\n\t\tlet s = slide || this.Reveal.getCurrentSlide();\n\t\tlet id = s ? s.getAttribute( 'id' ) : null;\n\t\tif( id ) {\n\t\t\tid = encodeURIComponent( id );\n\t\t}\n\n\t\tlet index = this.Reveal.getIndices( slide );\n\t\tif( !this.Reveal.getConfig().fragmentInURL ) {\n\t\t\tindex.f = undefined;\n\t\t}\n\n\t\t// If the current slide has an ID, use that as a named link,\n\t\t// but we don't support named links with a fragment index\n\t\tif( typeof id === 'string' && id.length ) {\n\t\t\turl = '/' + id;\n\n\t\t\t// If there is also a fragment, append that at the end\n\t\t\t// of the named link, like: #/named-link/2\n\t\t\tif( index.f >= 0 ) url += '/' + index.f;\n\t\t}\n\t\t// Otherwise use the /h/v index\n\t\telse {\n\t\t\tlet hashIndexBase = this.Reveal.getConfig().hashOneBasedIndex ? 1 : 0;\n\t\t\tif( index.h > 0 || index.v > 0 || index.f >= 0 ) url += index.h + hashIndexBase;\n\t\t\tif( index.v > 0 || index.f >= 0 ) url += '/' + (index.v + hashIndexBase );\n\t\t\tif( index.f >= 0 ) url += '/' + index.f;\n\t\t}\n\n\t\treturn url;\n\n\t}\n\n\t/**\n\t * Handler for the window level 'hashchange' event.\n\t *\n\t * @param {object} [event]\n\t */\n\tonWindowHashChange( event ) {\n\n\t\tthis.readURL();\n\n\t}\n\n}","import { queryAll } from '../utils/util.js'\nimport { isAndroid } from '../utils/device.js'\n\n/**\n * Manages our presentation controls. This includes both\n * the built-in control arrows as well as event monitoring\n * of any elements within the presentation with either of the\n * following helper classes:\n * - .navigate-up\n * - .navigate-right\n * - .navigate-down\n * - .navigate-left\n * - .navigate-next\n * - .navigate-prev\n */\nexport default class Controls {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t\tthis.onNavigateLeftClicked = this.onNavigateLeftClicked.bind( this );\n\t\tthis.onNavigateRightClicked = this.onNavigateRightClicked.bind( this );\n\t\tthis.onNavigateUpClicked = this.onNavigateUpClicked.bind( this );\n\t\tthis.onNavigateDownClicked = this.onNavigateDownClicked.bind( this );\n\t\tthis.onNavigatePrevClicked = this.onNavigatePrevClicked.bind( this );\n\t\tthis.onNavigateNextClicked = this.onNavigateNextClicked.bind( this );\n\n\t}\n\n\trender() {\n\n\t\tconst rtl = this.Reveal.getConfig().rtl;\n\t\tconst revealElement = this.Reveal.getRevealElement();\n\n\t\tthis.element = document.createElement( 'aside' );\n\t\tthis.element.className = 'controls';\n\t\tthis.element.innerHTML =\n\t\t\t`\n\t\t\t\n\t\t\t\n\t\t\t`;\n\n\t\tthis.Reveal.getRevealElement().appendChild( this.element );\n\n\t\t// There can be multiple instances of controls throughout the page\n\t\tthis.controlsLeft = queryAll( revealElement, '.navigate-left' );\n\t\tthis.controlsRight = queryAll( revealElement, '.navigate-right' );\n\t\tthis.controlsUp = queryAll( revealElement, '.navigate-up' );\n\t\tthis.controlsDown = queryAll( revealElement, '.navigate-down' );\n\t\tthis.controlsPrev = queryAll( revealElement, '.navigate-prev' );\n\t\tthis.controlsNext = queryAll( revealElement, '.navigate-next' );\n\n\t\t// The left, right and down arrows in the standard reveal.js controls\n\t\tthis.controlsRightArrow = this.element.querySelector( '.navigate-right' );\n\t\tthis.controlsLeftArrow = this.element.querySelector( '.navigate-left' );\n\t\tthis.controlsDownArrow = this.element.querySelector( '.navigate-down' );\n\n\t}\n\n\t/**\n\t * Called when the reveal.js config is updated.\n\t */\n\tconfigure( config, oldConfig ) {\n\n\t\tthis.element.style.display = config.controls ? 'block' : 'none';\n\n\t\tthis.element.setAttribute( 'data-controls-layout', config.controlsLayout );\n\t\tthis.element.setAttribute( 'data-controls-back-arrows', config.controlsBackArrows );\n\n\t}\n\n\tbind() {\n\n\t\t// Listen to both touch and click events, in case the device\n\t\t// supports both\n\t\tlet pointerEvents = [ 'touchstart', 'click' ];\n\n\t\t// Only support touch for Android, fixes double navigations in\n\t\t// stock browser\n\t\tif( isAndroid ) {\n\t\t\tpointerEvents = [ 'touchstart' ];\n\t\t}\n\n\t\tpointerEvents.forEach( eventName => {\n\t\t\tthis.controlsLeft.forEach( el => el.addEventListener( eventName, this.onNavigateLeftClicked, false ) );\n\t\t\tthis.controlsRight.forEach( el => el.addEventListener( eventName, this.onNavigateRightClicked, false ) );\n\t\t\tthis.controlsUp.forEach( el => el.addEventListener( eventName, this.onNavigateUpClicked, false ) );\n\t\t\tthis.controlsDown.forEach( el => el.addEventListener( eventName, this.onNavigateDownClicked, false ) );\n\t\t\tthis.controlsPrev.forEach( el => el.addEventListener( eventName, this.onNavigatePrevClicked, false ) );\n\t\t\tthis.controlsNext.forEach( el => el.addEventListener( eventName, this.onNavigateNextClicked, false ) );\n\t\t} );\n\n\t}\n\n\tunbind() {\n\n\t\t[ 'touchstart', 'click' ].forEach( eventName => {\n\t\t\tthis.controlsLeft.forEach( el => el.removeEventListener( eventName, this.onNavigateLeftClicked, false ) );\n\t\t\tthis.controlsRight.forEach( el => el.removeEventListener( eventName, this.onNavigateRightClicked, false ) );\n\t\t\tthis.controlsUp.forEach( el => el.removeEventListener( eventName, this.onNavigateUpClicked, false ) );\n\t\t\tthis.controlsDown.forEach( el => el.removeEventListener( eventName, this.onNavigateDownClicked, false ) );\n\t\t\tthis.controlsPrev.forEach( el => el.removeEventListener( eventName, this.onNavigatePrevClicked, false ) );\n\t\t\tthis.controlsNext.forEach( el => el.removeEventListener( eventName, this.onNavigateNextClicked, false ) );\n\t\t} );\n\n\t}\n\n\t/**\n\t * Updates the state of all control/navigation arrows.\n\t */\n\tupdate() {\n\n\t\tlet routes = this.Reveal.availableRoutes();\n\n\t\t// Remove the 'enabled' class from all directions\n\t\t[...this.controlsLeft, ...this.controlsRight, ...this.controlsUp, ...this.controlsDown, ...this.controlsPrev, ...this.controlsNext].forEach( node => {\n\t\t\tnode.classList.remove( 'enabled', 'fragmented' );\n\n\t\t\t// Set 'disabled' attribute on all directions\n\t\t\tnode.setAttribute( 'disabled', 'disabled' );\n\t\t} );\n\n\t\t// Add the 'enabled' class to the available routes; remove 'disabled' attribute to enable buttons\n\t\tif( routes.left ) this.controlsLeft.forEach( el => { el.classList.add( 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\t\tif( routes.right ) this.controlsRight.forEach( el => { el.classList.add( 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\t\tif( routes.up ) this.controlsUp.forEach( el => { el.classList.add( 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\t\tif( routes.down ) this.controlsDown.forEach( el => { el.classList.add( 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\n\t\t// Prev/next buttons\n\t\tif( routes.left || routes.up ) this.controlsPrev.forEach( el => { el.classList.add( 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\t\tif( routes.right || routes.down ) this.controlsNext.forEach( el => { el.classList.add( 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\n\t\t// Highlight fragment directions\n\t\tlet currentSlide = this.Reveal.getCurrentSlide();\n\t\tif( currentSlide ) {\n\n\t\t\tlet fragmentsRoutes = this.Reveal.fragments.availableRoutes();\n\n\t\t\t// Always apply fragment decorator to prev/next buttons\n\t\t\tif( fragmentsRoutes.prev ) this.controlsPrev.forEach( el => { el.classList.add( 'fragmented', 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\t\t\tif( fragmentsRoutes.next ) this.controlsNext.forEach( el => { el.classList.add( 'fragmented', 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\n\t\t\t// Apply fragment decorators to directional buttons based on\n\t\t\t// what slide axis they are in\n\t\t\tif( this.Reveal.isVerticalSlide( currentSlide ) ) {\n\t\t\t\tif( fragmentsRoutes.prev ) this.controlsUp.forEach( el => { el.classList.add( 'fragmented', 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\t\t\t\tif( fragmentsRoutes.next ) this.controlsDown.forEach( el => { el.classList.add( 'fragmented', 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\t\t\t}\n\t\t\telse {\n\t\t\t\tif( fragmentsRoutes.prev ) this.controlsLeft.forEach( el => { el.classList.add( 'fragmented', 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\t\t\t\tif( fragmentsRoutes.next ) this.controlsRight.forEach( el => { el.classList.add( 'fragmented', 'enabled' ); el.removeAttribute( 'disabled' ); } );\n\t\t\t}\n\n\t\t}\n\n\t\tif( this.Reveal.getConfig().controlsTutorial ) {\n\n\t\t\tlet indices = this.Reveal.getIndices();\n\n\t\t\t// Highlight control arrows with an animation to ensure\n\t\t\t// that the viewer knows how to navigate\n\t\t\tif( !this.Reveal.hasNavigatedVertically() && routes.down ) {\n\t\t\t\tthis.controlsDownArrow.classList.add( 'highlight' );\n\t\t\t}\n\t\t\telse {\n\t\t\t\tthis.controlsDownArrow.classList.remove( 'highlight' );\n\n\t\t\t\tif( this.Reveal.getConfig().rtl ) {\n\n\t\t\t\t\tif( !this.Reveal.hasNavigatedHorizontally() && routes.left && indices.v === 0 ) {\n\t\t\t\t\t\tthis.controlsLeftArrow.classList.add( 'highlight' );\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tthis.controlsLeftArrow.classList.remove( 'highlight' );\n\t\t\t\t\t}\n\n\t\t\t\t} else {\n\n\t\t\t\t\tif( !this.Reveal.hasNavigatedHorizontally() && routes.right && indices.v === 0 ) {\n\t\t\t\t\t\tthis.controlsRightArrow.classList.add( 'highlight' );\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tthis.controlsRightArrow.classList.remove( 'highlight' );\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\tdestroy() {\n\n\t\tthis.unbind();\n\t\tthis.element.remove();\n\n\t}\n\n\t/**\n\t * Event handlers for navigation control buttons.\n\t */\n\tonNavigateLeftClicked( event ) {\n\n\t\tevent.preventDefault();\n\t\tthis.Reveal.onUserInput();\n\n\t\tif( this.Reveal.getConfig().navigationMode === 'linear' ) {\n\t\t\tthis.Reveal.prev();\n\t\t}\n\t\telse {\n\t\t\tthis.Reveal.left();\n\t\t}\n\n\t}\n\n\tonNavigateRightClicked( event ) {\n\n\t\tevent.preventDefault();\n\t\tthis.Reveal.onUserInput();\n\n\t\tif( this.Reveal.getConfig().navigationMode === 'linear' ) {\n\t\t\tthis.Reveal.next();\n\t\t}\n\t\telse {\n\t\t\tthis.Reveal.right();\n\t\t}\n\n\t}\n\n\tonNavigateUpClicked( event ) {\n\n\t\tevent.preventDefault();\n\t\tthis.Reveal.onUserInput();\n\n\t\tthis.Reveal.up();\n\n\t}\n\n\tonNavigateDownClicked( event ) {\n\n\t\tevent.preventDefault();\n\t\tthis.Reveal.onUserInput();\n\n\t\tthis.Reveal.down();\n\n\t}\n\n\tonNavigatePrevClicked( event ) {\n\n\t\tevent.preventDefault();\n\t\tthis.Reveal.onUserInput();\n\n\t\tthis.Reveal.prev();\n\n\t}\n\n\tonNavigateNextClicked( event ) {\n\n\t\tevent.preventDefault();\n\t\tthis.Reveal.onUserInput();\n\n\t\tthis.Reveal.next();\n\n\t}\n\n\n}","/**\n * Creates a visual progress bar for the presentation.\n */\nexport default class Progress {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t\tthis.onProgressClicked = this.onProgressClicked.bind( this );\n\n\t}\n\n\trender() {\n\n\t\tthis.element = document.createElement( 'div' );\n\t\tthis.element.className = 'progress';\n\t\tthis.Reveal.getRevealElement().appendChild( this.element );\n\n\t\tthis.bar = document.createElement( 'span' );\n\t\tthis.element.appendChild( this.bar );\n\n\t}\n\n\t/**\n\t * Called when the reveal.js config is updated.\n\t */\n\tconfigure( config, oldConfig ) {\n\n\t\tthis.element.style.display = config.progress ? 'block' : 'none';\n\n\t}\n\n\tbind() {\n\n\t\tif( this.Reveal.getConfig().progress && this.element ) {\n\t\t\tthis.element.addEventListener( 'click', this.onProgressClicked, false );\n\t\t}\n\n\t}\n\n\tunbind() {\n\n\t\tif ( this.Reveal.getConfig().progress && this.element ) {\n\t\t\tthis.element.removeEventListener( 'click', this.onProgressClicked, false );\n\t\t}\n\n\t}\n\n\t/**\n\t * Updates the progress bar to reflect the current slide.\n\t */\n\tupdate() {\n\n\t\t// Update progress if enabled\n\t\tif( this.Reveal.getConfig().progress && this.bar ) {\n\n\t\t\tlet scale = this.Reveal.getProgress();\n\n\t\t\t// Don't fill the progress bar if there's only one slide\n\t\t\tif( this.Reveal.getTotalSlides() < 2 ) {\n\t\t\t\tscale = 0;\n\t\t\t}\n\n\t\t\tthis.bar.style.transform = 'scaleX('+ scale +')';\n\n\t\t}\n\n\t}\n\n\tgetMaxWidth() {\n\n\t\treturn this.Reveal.getRevealElement().offsetWidth;\n\n\t}\n\n\t/**\n\t * Clicking on the progress bar results in a navigation to the\n\t * closest approximate horizontal slide using this equation:\n\t *\n\t * ( clickX / presentationWidth ) * numberOfSlides\n\t *\n\t * @param {object} event\n\t */\n\tonProgressClicked( event ) {\n\n\t\tthis.Reveal.onUserInput( event );\n\n\t\tevent.preventDefault();\n\n\t\tlet slides = this.Reveal.getSlides();\n\t\tlet slidesTotal = slides.length;\n\t\tlet slideIndex = Math.floor( ( event.clientX / this.getMaxWidth() ) * slidesTotal );\n\n\t\tif( this.Reveal.getConfig().rtl ) {\n\t\t\tslideIndex = slidesTotal - slideIndex;\n\t\t}\n\n\t\tlet targetIndices = this.Reveal.getIndices(slides[slideIndex]);\n\t\tthis.Reveal.slide( targetIndices.h, targetIndices.v );\n\n\t}\n\n\tdestroy() {\n\n\t\tthis.element.remove();\n\n\t}\n\n}","/**\n * Handles hiding of the pointer/cursor when inactive.\n */\nexport default class Pointer {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t\t// Throttles mouse wheel navigation\n\t\tthis.lastMouseWheelStep = 0;\n\n\t\t// Is the mouse pointer currently hidden from view\n\t\tthis.cursorHidden = false;\n\n\t\t// Timeout used to determine when the cursor is inactive\n\t\tthis.cursorInactiveTimeout = 0;\n\n\t\tthis.onDocumentCursorActive = this.onDocumentCursorActive.bind( this );\n\t\tthis.onDocumentMouseScroll = this.onDocumentMouseScroll.bind( this );\n\n\t}\n\n\t/**\n\t * Called when the reveal.js config is updated.\n\t */\n\tconfigure( config, oldConfig ) {\n\n\t\tif( config.mouseWheel ) {\n\t\t\tdocument.addEventListener( 'DOMMouseScroll', this.onDocumentMouseScroll, false ); // FF\n\t\t\tdocument.addEventListener( 'mousewheel', this.onDocumentMouseScroll, false );\n\t\t}\n\t\telse {\n\t\t\tdocument.removeEventListener( 'DOMMouseScroll', this.onDocumentMouseScroll, false ); // FF\n\t\t\tdocument.removeEventListener( 'mousewheel', this.onDocumentMouseScroll, false );\n\t\t}\n\n\t\t// Auto-hide the mouse pointer when its inactive\n\t\tif( config.hideInactiveCursor ) {\n\t\t\tdocument.addEventListener( 'mousemove', this.onDocumentCursorActive, false );\n\t\t\tdocument.addEventListener( 'mousedown', this.onDocumentCursorActive, false );\n\t\t}\n\t\telse {\n\t\t\tthis.showCursor();\n\n\t\t\tdocument.removeEventListener( 'mousemove', this.onDocumentCursorActive, false );\n\t\t\tdocument.removeEventListener( 'mousedown', this.onDocumentCursorActive, false );\n\t\t}\n\n\t}\n\n\t/**\n\t * Shows the mouse pointer after it has been hidden with\n\t * #hideCursor.\n\t */\n\tshowCursor() {\n\n\t\tif( this.cursorHidden ) {\n\t\t\tthis.cursorHidden = false;\n\t\t\tthis.Reveal.getRevealElement().style.cursor = '';\n\t\t}\n\n\t}\n\n\t/**\n\t * Hides the mouse pointer when it's on top of the .reveal\n\t * container.\n\t */\n\thideCursor() {\n\n\t\tif( this.cursorHidden === false ) {\n\t\t\tthis.cursorHidden = true;\n\t\t\tthis.Reveal.getRevealElement().style.cursor = 'none';\n\t\t}\n\n\t}\n\n\tdestroy() {\n\n\t\tthis.showCursor();\n\n\t\tdocument.removeEventListener( 'DOMMouseScroll', this.onDocumentMouseScroll, false );\n\t\tdocument.removeEventListener( 'mousewheel', this.onDocumentMouseScroll, false );\n\t\tdocument.removeEventListener( 'mousemove', this.onDocumentCursorActive, false );\n\t\tdocument.removeEventListener( 'mousedown', this.onDocumentCursorActive, false );\n\n\t}\n\n\t/**\n\t * Called whenever there is mouse input at the document level\n\t * to determine if the cursor is active or not.\n\t *\n\t * @param {object} event\n\t */\n\tonDocumentCursorActive( event ) {\n\n\t\tthis.showCursor();\n\n\t\tclearTimeout( this.cursorInactiveTimeout );\n\n\t\tthis.cursorInactiveTimeout = setTimeout( this.hideCursor.bind( this ), this.Reveal.getConfig().hideCursorTime );\n\n\t}\n\n\t/**\n\t * Handles mouse wheel scrolling, throttled to avoid skipping\n\t * multiple slides.\n\t *\n\t * @param {object} event\n\t */\n\tonDocumentMouseScroll( event ) {\n\n\t\tif( Date.now() - this.lastMouseWheelStep > 1000 ) {\n\n\t\t\tthis.lastMouseWheelStep = Date.now();\n\n\t\t\tlet delta = event.detail || -event.wheelDelta;\n\t\t\tif( delta > 0 ) {\n\t\t\t\tthis.Reveal.next();\n\t\t\t}\n\t\t\telse if( delta < 0 ) {\n\t\t\t\tthis.Reveal.prev();\n\t\t\t}\n\n\t\t}\n\n\t}\n\n}","/**\n * Loads a JavaScript file from the given URL and executes it.\n *\n * @param {string} url Address of the .js file to load\n * @param {function} callback Method to invoke when the script\n * has loaded and executed\n */\nexport const loadScript = ( url, callback ) => {\n\n\tconst script = document.createElement( 'script' );\n\tscript.type = 'text/javascript';\n\tscript.async = false;\n\tscript.defer = false;\n\tscript.src = url;\n\n\tif( typeof callback === 'function' ) {\n\n\t\t// Success callback\n\t\tscript.onload = script.onreadystatechange = event => {\n\t\t\tif( event.type === 'load' || /loaded|complete/.test( script.readyState ) ) {\n\n\t\t\t\t// Kill event listeners\n\t\t\t\tscript.onload = script.onreadystatechange = script.onerror = null;\n\n\t\t\t\tcallback();\n\n\t\t\t}\n\t\t};\n\n\t\t// Error callback\n\t\tscript.onerror = err => {\n\n\t\t\t// Kill event listeners\n\t\t\tscript.onload = script.onreadystatechange = script.onerror = null;\n\n\t\t\tcallback( new Error( 'Failed loading script: ' + script.src + '\\n' + err ) );\n\n\t\t};\n\n\t}\n\n\t// Append the script at the end of \n\tconst head = document.querySelector( 'head' );\n\thead.insertBefore( script, head.lastChild );\n\n}","import { loadScript } from '../utils/loader.js'\n\n/**\n * Manages loading and registering of reveal.js plugins.\n */\nexport default class Plugins {\n\n\tconstructor( reveal ) {\n\n\t\tthis.Reveal = reveal;\n\n\t\t// Flags our current state (idle -> loading -> loaded)\n\t\tthis.state = 'idle';\n\n\t\t// An id:instance map of currently registed plugins\n\t\tthis.registeredPlugins = {};\n\n\t\tthis.asyncDependencies = [];\n\n\t}\n\n\t/**\n\t * Loads reveal.js dependencies, registers and\n\t * initializes plugins.\n\t *\n\t * Plugins are direct references to a reveal.js plugin\n\t * object that we register and initialize after any\n\t * synchronous dependencies have loaded.\n\t *\n\t * Dependencies are defined via the 'dependencies' config\n\t * option and will be loaded prior to starting reveal.js.\n\t * Some dependencies may have an 'async' flag, if so they\n\t * will load after reveal.js has been started up.\n\t */\n\tload( plugins, dependencies ) {\n\n\t\tthis.state = 'loading';\n\n\t\tplugins.forEach( this.registerPlugin.bind( this ) );\n\n\t\treturn new Promise( resolve => {\n\n\t\t\tlet scripts = [],\n\t\t\t\tscriptsToLoad = 0;\n\n\t\t\tdependencies.forEach( s => {\n\t\t\t\t// Load if there's no condition or the condition is truthy\n\t\t\t\tif( !s.condition || s.condition() ) {\n\t\t\t\t\tif( s.async ) {\n\t\t\t\t\t\tthis.asyncDependencies.push( s );\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tscripts.push( s );\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t} );\n\n\t\t\tif( scripts.length ) {\n\t\t\t\tscriptsToLoad = scripts.length;\n\n\t\t\t\tconst scriptLoadedCallback = (s) => {\n\t\t\t\t\tif( s && typeof s.callback === 'function' ) s.callback();\n\n\t\t\t\t\tif( --scriptsToLoad === 0 ) {\n\t\t\t\t\t\tthis.initPlugins().then( resolve );\n\t\t\t\t\t}\n\t\t\t\t};\n\n\t\t\t\t// Load synchronous scripts\n\t\t\t\tscripts.forEach( s => {\n\t\t\t\t\tif( typeof s.id === 'string' ) {\n\t\t\t\t\t\tthis.registerPlugin( s );\n\t\t\t\t\t\tscriptLoadedCallback( s );\n\t\t\t\t\t}\n\t\t\t\t\telse if( typeof s.src === 'string' ) {\n\t\t\t\t\t\tloadScript( s.src, () => scriptLoadedCallback(s) );\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tconsole.warn( 'Unrecognized plugin format', s );\n\t\t\t\t\t\tscriptLoadedCallback();\n\t\t\t\t\t}\n\t\t\t\t} );\n\t\t\t}\n\t\t\telse {\n\t\t\t\tthis.initPlugins().then( resolve );\n\t\t\t}\n\n\t\t} );\n\n\t}\n\n\t/**\n\t * Initializes our plugins and waits for them to be ready\n\t * before proceeding.\n\t */\n\tinitPlugins() {\n\n\t\treturn new Promise( resolve => {\n\n\t\t\tlet pluginValues = Object.values( this.registeredPlugins );\n\t\t\tlet pluginsToInitialize = pluginValues.length;\n\n\t\t\t// If there are no plugins, skip this step\n\t\t\tif( pluginsToInitialize === 0 ) {\n\t\t\t\tthis.loadAsync().then( resolve );\n\t\t\t}\n\t\t\t// ... otherwise initialize plugins\n\t\t\telse {\n\n\t\t\t\tlet initNextPlugin;\n\n\t\t\t\tlet afterPlugInitialized = () => {\n\t\t\t\t\tif( --pluginsToInitialize === 0 ) {\n\t\t\t\t\t\tthis.loadAsync().then( resolve );\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tinitNextPlugin();\n\t\t\t\t\t}\n\t\t\t\t};\n\n\t\t\t\tlet i = 0;\n\n\t\t\t\t// Initialize plugins serially\n\t\t\t\tinitNextPlugin = () => {\n\n\t\t\t\t\tlet plugin = pluginValues[i++];\n\n\t\t\t\t\t// If the plugin has an 'init' method, invoke it\n\t\t\t\t\tif( typeof plugin.init === 'function' ) {\n\t\t\t\t\t\tlet promise = plugin.init( this.Reveal );\n\n\t\t\t\t\t\t// If the plugin returned a Promise, wait for it\n\t\t\t\t\t\tif( promise && typeof promise.then === 'function' ) {\n\t\t\t\t\t\t\tpromise.then( afterPlugInitialized );\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tafterPlugInitialized();\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tafterPlugInitialized();\n\t\t\t\t\t}\n\n\t\t\t\t}\n\n\t\t\t\tinitNextPlugin();\n\n\t\t\t}\n\n\t\t} )\n\n\t}\n\n\t/**\n\t * Loads all async reveal.js dependencies.\n\t */\n\tloadAsync() {\n\n\t\tthis.state = 'loaded';\n\n\t\tif( this.asyncDependencies.length ) {\n\t\t\tthis.asyncDependencies.forEach( s => {\n\t\t\t\tloadScript( s.src, s.callback );\n\t\t\t} );\n\t\t}\n\n\t\treturn Promise.resolve();\n\n\t}\n\n\t/**\n\t * Registers a new plugin with this reveal.js instance.\n\t *\n\t * reveal.js waits for all regisered plugins to initialize\n\t * before considering itself ready, as long as the plugin\n\t * is registered before calling `Reveal.initialize()`.\n\t */\n\tregisterPlugin( plugin ) {\n\n\t\t// Backwards compatibility to make reveal.js ~3.9.0\n\t\t// plugins work with reveal.js 4.0.0\n\t\tif( arguments.length === 2 && typeof arguments[0] === 'string' ) {\n\t\t\tplugin = arguments[1];\n\t\t\tplugin.id = arguments[0];\n\t\t}\n\t\t// Plugin can optionally be a function which we call\n\t\t// to create an instance of the plugin\n\t\telse if( typeof plugin === 'function' ) {\n\t\t\tplugin = plugin();\n\t\t}\n\n\t\tlet id = plugin.id;\n\n\t\tif( typeof id !== 'string' ) {\n\t\t\tconsole.warn( 'Unrecognized plugin format; can\\'t find plugin.id', plugin );\n\t\t}\n\t\telse if( this.registeredPlugins[id] === undefined ) {\n\t\t\tthis.registeredPlugins[id] = plugin;\n\n\t\t\t// If a plugin is registered after reveal.js is loaded,\n\t\t\t// initialize it right away\n\t\t\tif( this.state === 'loaded' && typeof plugin.init === 'function' ) {\n\t\t\t\tplugin.init( this.Reveal );\n\t\t\t}\n\t\t}\n\t\telse {\n\t\t\tconsole.warn( 'reveal.js: \"'+ id +'\" plugin has already been registered' );\n\t\t}\n\n\t}\n\n\t/**\n\t * Checks if a specific plugin has been registered.\n\t *\n\t * @param {String} id Unique plugin identifier\n\t */\n\thasPlugin( id ) {\n\n\t\treturn !!this.registeredPlugins[id];\n\n\t}\n\n\t/**\n\t * Returns the specific plugin instance, if a plugin\n\t * with the given ID has been registered.\n\t *\n\t * @param {String} id Unique plugin identifier\n\t */\n\tgetPlugin( id ) {\n\n\t\treturn this.registeredPlugins[id];\n\n\t}\n\n\tgetRegisteredPlugins() {\n\n\t\treturn this.registeredPlugins;\n\n\t}\n\n\tdestroy() {\n\n\t\tObject.values( this.registeredPlugins ).forEach( plugin => {\n\t\t\tif( typeof plugin.destroy === 'function' ) {\n\t\t\t\tplugin.destroy();\n\t\t\t}\n\t\t} );\n\n\t\tthis.registeredPlugins = {};\n\t\tthis.asyncDependencies = [];\n\n\t}\n\n}\n","import { SLIDES_SELECTOR } from '../utils/constants.js'\nimport { queryAll, createStyleSheet } from '../utils/util.js'\n\n/**\n * Setups up our presentation for printing/exporting to PDF.\n */\nexport default class Print {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t}\n\n\t/**\n\t * Configures the presentation for printing to a static\n\t * PDF.\n\t */\n\tasync setupPDF() {\n\n\t\tconst config = this.Reveal.getConfig();\n\t\tconst slides = queryAll( this.Reveal.getRevealElement(), SLIDES_SELECTOR )\n\n\t\t// Compute slide numbers now, before we start duplicating slides\n\t\tconst doingSlideNumbers = config.slideNumber && /all|print/i.test( config.showSlideNumber );\n\n\t\tconst slideSize = this.Reveal.getComputedSlideSize( window.innerWidth, window.innerHeight );\n\n\t\t// Dimensions of the PDF pages\n\t\tconst pageWidth = Math.floor( slideSize.width * ( 1 + config.margin ) ),\n\t\t\tpageHeight = Math.floor( slideSize.height * ( 1 + config.margin ) );\n\n\t\t// Dimensions of slides within the pages\n\t\tconst slideWidth = slideSize.width,\n\t\t\tslideHeight = slideSize.height;\n\n\t\tawait new Promise( requestAnimationFrame );\n\n\t\t// Let the browser know what page size we want to print\n\t\tcreateStyleSheet( '@page{size:'+ pageWidth +'px '+ pageHeight +'px; margin: 0px;}' );\n\n\t\t// Limit the size of certain elements to the dimensions of the slide\n\t\tcreateStyleSheet( '.reveal section>img, .reveal section>video, .reveal section>iframe{max-width: '+ slideWidth +'px; max-height:'+ slideHeight +'px}' );\n\n\t\tdocument.documentElement.classList.add( 'print-pdf' );\n\t\tdocument.body.style.width = pageWidth + 'px';\n\t\tdocument.body.style.height = pageHeight + 'px';\n\n\t\tconst viewportElement = document.querySelector( '.reveal-viewport' );\n\t\tlet presentationBackground;\n\t\tif( viewportElement ) {\n\t\t\tconst viewportStyles = window.getComputedStyle( viewportElement );\n\t\t\tif( viewportStyles && viewportStyles.background ) {\n\t\t\t\tpresentationBackground = viewportStyles.background;\n\t\t\t}\n\t\t}\n\n\t\t// Make sure stretch elements fit on slide\n\t\tawait new Promise( requestAnimationFrame );\n\t\tthis.Reveal.layoutSlideContents( slideWidth, slideHeight );\n\n\t\t// Batch scrollHeight access to prevent layout thrashing\n\t\tawait new Promise( requestAnimationFrame );\n\n\t\tconst slideScrollHeights = slides.map( slide => slide.scrollHeight );\n\n\t\tconst pages = [];\n\t\tconst pageContainer = slides[0].parentNode;\n\n\t\t// Slide and slide background layout\n\t\tslides.forEach( function( slide, index ) {\n\n\t\t\t// Vertical stacks are not centred since their section\n\t\t\t// children will be\n\t\t\tif( slide.classList.contains( 'stack' ) === false ) {\n\t\t\t\t// Center the slide inside of the page, giving the slide some margin\n\t\t\t\tlet left = ( pageWidth - slideWidth ) / 2;\n\t\t\t\tlet top = ( pageHeight - slideHeight ) / 2;\n\n\t\t\t\tconst contentHeight = slideScrollHeights[ index ];\n\t\t\t\tlet numberOfPages = Math.max( Math.ceil( contentHeight / pageHeight ), 1 );\n\n\t\t\t\t// Adhere to configured pages per slide limit\n\t\t\t\tnumberOfPages = Math.min( numberOfPages, config.pdfMaxPagesPerSlide );\n\n\t\t\t\t// Center slides vertically\n\t\t\t\tif( numberOfPages === 1 && config.center || slide.classList.contains( 'center' ) ) {\n\t\t\t\t\ttop = Math.max( ( pageHeight - contentHeight ) / 2, 0 );\n\t\t\t\t}\n\n\t\t\t\t// Wrap the slide in a page element and hide its overflow\n\t\t\t\t// so that no page ever flows onto another\n\t\t\t\tconst page = document.createElement( 'div' );\n\t\t\t\tpages.push( page );\n\n\t\t\t\tpage.className = 'pdf-page';\n\t\t\t\tpage.style.height = ( ( pageHeight + config.pdfPageHeightOffset ) * numberOfPages ) + 'px';\n\n\t\t\t\t// Copy the presentation-wide background to each individual\n\t\t\t\t// page when printing\n\t\t\t\tif( presentationBackground ) {\n\t\t\t\t\tpage.style.background = presentationBackground;\n\t\t\t\t}\n\n\t\t\t\tpage.appendChild( slide );\n\n\t\t\t\t// Position the slide inside of the page\n\t\t\t\tslide.style.left = left + 'px';\n\t\t\t\tslide.style.top = top + 'px';\n\t\t\t\tslide.style.width = slideWidth + 'px';\n\n\t\t\t\t// Re-run the slide layout so that r-fit-text is applied based on\n\t\t\t\t// the printed slide size\n\t\t\t\tthis.Reveal.slideContent.layout( slide )\n\n\t\t\t\tif( slide.slideBackgroundElement ) {\n\t\t\t\t\tpage.insertBefore( slide.slideBackgroundElement, slide );\n\t\t\t\t}\n\n\t\t\t\t// Inject notes if `showNotes` is enabled\n\t\t\t\tif( config.showNotes ) {\n\n\t\t\t\t\t// Are there notes for this slide?\n\t\t\t\t\tconst notes = this.Reveal.getSlideNotes( slide );\n\t\t\t\t\tif( notes ) {\n\n\t\t\t\t\t\tconst notesSpacing = 8;\n\t\t\t\t\t\tconst notesLayout = typeof config.showNotes === 'string' ? config.showNotes : 'inline';\n\t\t\t\t\t\tconst notesElement = document.createElement( 'div' );\n\t\t\t\t\t\tnotesElement.classList.add( 'speaker-notes' );\n\t\t\t\t\t\tnotesElement.classList.add( 'speaker-notes-pdf' );\n\t\t\t\t\t\tnotesElement.setAttribute( 'data-layout', notesLayout );\n\t\t\t\t\t\tnotesElement.innerHTML = notes;\n\n\t\t\t\t\t\tif( notesLayout === 'separate-page' ) {\n\t\t\t\t\t\t\tpages.push( notesElement );\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tnotesElement.style.left = notesSpacing + 'px';\n\t\t\t\t\t\t\tnotesElement.style.bottom = notesSpacing + 'px';\n\t\t\t\t\t\t\tnotesElement.style.width = ( pageWidth - notesSpacing*2 ) + 'px';\n\t\t\t\t\t\t\tpage.appendChild( notesElement );\n\t\t\t\t\t\t}\n\n\t\t\t\t\t}\n\n\t\t\t\t}\n\n\t\t\t\t// Inject slide numbers if `slideNumbers` are enabled\n\t\t\t\tif( doingSlideNumbers ) {\n\t\t\t\t\tconst slideNumber = index + 1;\n\t\t\t\t\tconst numberElement = document.createElement( 'div' );\n\t\t\t\t\tnumberElement.classList.add( 'slide-number' );\n\t\t\t\t\tnumberElement.classList.add( 'slide-number-pdf' );\n\t\t\t\t\tnumberElement.innerHTML = slideNumber;\n\t\t\t\t\tpage.appendChild( numberElement );\n\t\t\t\t}\n\n\t\t\t\t// Copy page and show fragments one after another\n\t\t\t\tif( config.pdfSeparateFragments ) {\n\n\t\t\t\t\t// Each fragment 'group' is an array containing one or more\n\t\t\t\t\t// fragments. Multiple fragments that appear at the same time\n\t\t\t\t\t// are part of the same group.\n\t\t\t\t\tconst fragmentGroups = this.Reveal.fragments.sort( page.querySelectorAll( '.fragment' ), true );\n\n\t\t\t\t\tlet previousFragmentStep;\n\n\t\t\t\t\tfragmentGroups.forEach( function( fragments ) {\n\n\t\t\t\t\t\t// Remove 'current-fragment' from the previous group\n\t\t\t\t\t\tif( previousFragmentStep ) {\n\t\t\t\t\t\t\tpreviousFragmentStep.forEach( function( fragment ) {\n\t\t\t\t\t\t\t\tfragment.classList.remove( 'current-fragment' );\n\t\t\t\t\t\t\t} );\n\t\t\t\t\t\t}\n\n\t\t\t\t\t\t// Show the fragments for the current index\n\t\t\t\t\t\tfragments.forEach( function( fragment ) {\n\t\t\t\t\t\t\tfragment.classList.add( 'visible', 'current-fragment' );\n\t\t\t\t\t\t}, this );\n\n\t\t\t\t\t\t// Create a separate page for the current fragment state\n\t\t\t\t\t\tconst clonedPage = page.cloneNode( true );\n\t\t\t\t\t\tpages.push( clonedPage );\n\n\t\t\t\t\t\tpreviousFragmentStep = fragments;\n\n\t\t\t\t\t}, this );\n\n\t\t\t\t\t// Reset the first/original page so that all fragments are hidden\n\t\t\t\t\tfragmentGroups.forEach( function( fragments ) {\n\t\t\t\t\t\tfragments.forEach( function( fragment ) {\n\t\t\t\t\t\t\tfragment.classList.remove( 'visible', 'current-fragment' );\n\t\t\t\t\t\t} );\n\t\t\t\t\t} );\n\n\t\t\t\t}\n\t\t\t\t// Show all fragments\n\t\t\t\telse {\n\t\t\t\t\tqueryAll( page, '.fragment:not(.fade-out)' ).forEach( function( fragment ) {\n\t\t\t\t\t\tfragment.classList.add( 'visible' );\n\t\t\t\t\t} );\n\t\t\t\t}\n\n\t\t\t}\n\n\t\t}, this );\n\n\t\tawait new Promise( requestAnimationFrame );\n\n\t\tpages.forEach( page => pageContainer.appendChild( page ) );\n\n\t\t// Notify subscribers that the PDF layout is good to go\n\t\tthis.Reveal.dispatchEvent({ type: 'pdf-ready' });\n\n\t}\n\n\t/**\n\t * Checks if this instance is being used to print a PDF.\n\t */\n\tisPrintingPDF() {\n\n\t\treturn ( /print-pdf/gi ).test( window.location.search );\n\n\t}\n\n}\n","import { isAndroid } from '../utils/device.js'\nimport { matches } from '../utils/util.js'\n\nconst SWIPE_THRESHOLD = 40;\n\n/**\n * Controls all touch interactions and navigations for\n * a presentation.\n */\nexport default class Touch {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t\t// Holds information about the currently ongoing touch interaction\n\t\tthis.touchStartX = 0;\n\t\tthis.touchStartY = 0;\n\t\tthis.touchStartCount = 0;\n\t\tthis.touchCaptured = false;\n\n\t\tthis.onPointerDown = this.onPointerDown.bind( this );\n\t\tthis.onPointerMove = this.onPointerMove.bind( this );\n\t\tthis.onPointerUp = this.onPointerUp.bind( this );\n\t\tthis.onTouchStart = this.onTouchStart.bind( this );\n\t\tthis.onTouchMove = this.onTouchMove.bind( this );\n\t\tthis.onTouchEnd = this.onTouchEnd.bind( this );\n\n\t}\n\n\t/**\n\t *\n\t */\n\tbind() {\n\n\t\tlet revealElement = this.Reveal.getRevealElement();\n\n\t\tif( 'onpointerdown' in window ) {\n\t\t\t// Use W3C pointer events\n\t\t\trevealElement.addEventListener( 'pointerdown', this.onPointerDown, false );\n\t\t\trevealElement.addEventListener( 'pointermove', this.onPointerMove, false );\n\t\t\trevealElement.addEventListener( 'pointerup', this.onPointerUp, false );\n\t\t}\n\t\telse if( window.navigator.msPointerEnabled ) {\n\t\t\t// IE 10 uses prefixed version of pointer events\n\t\t\trevealElement.addEventListener( 'MSPointerDown', this.onPointerDown, false );\n\t\t\trevealElement.addEventListener( 'MSPointerMove', this.onPointerMove, false );\n\t\t\trevealElement.addEventListener( 'MSPointerUp', this.onPointerUp, false );\n\t\t}\n\t\telse {\n\t\t\t// Fall back to touch events\n\t\t\trevealElement.addEventListener( 'touchstart', this.onTouchStart, false );\n\t\t\trevealElement.addEventListener( 'touchmove', this.onTouchMove, false );\n\t\t\trevealElement.addEventListener( 'touchend', this.onTouchEnd, false );\n\t\t}\n\n\t}\n\n\t/**\n\t *\n\t */\n\tunbind() {\n\n\t\tlet revealElement = this.Reveal.getRevealElement();\n\n\t\trevealElement.removeEventListener( 'pointerdown', this.onPointerDown, false );\n\t\trevealElement.removeEventListener( 'pointermove', this.onPointerMove, false );\n\t\trevealElement.removeEventListener( 'pointerup', this.onPointerUp, false );\n\n\t\trevealElement.removeEventListener( 'MSPointerDown', this.onPointerDown, false );\n\t\trevealElement.removeEventListener( 'MSPointerMove', this.onPointerMove, false );\n\t\trevealElement.removeEventListener( 'MSPointerUp', this.onPointerUp, false );\n\n\t\trevealElement.removeEventListener( 'touchstart', this.onTouchStart, false );\n\t\trevealElement.removeEventListener( 'touchmove', this.onTouchMove, false );\n\t\trevealElement.removeEventListener( 'touchend', this.onTouchEnd, false );\n\n\t}\n\n\t/**\n\t * Checks if the target element prevents the triggering of\n\t * swipe navigation.\n\t */\n\tisSwipePrevented( target ) {\n\n\t\t// Prevent accidental swipes when scrubbing timelines\n\t\tif( matches( target, 'video, audio' ) ) return true;\n\n\t\twhile( target && typeof target.hasAttribute === 'function' ) {\n\t\t\tif( target.hasAttribute( 'data-prevent-swipe' ) ) return true;\n\t\t\ttarget = target.parentNode;\n\t\t}\n\n\t\treturn false;\n\n\t}\n\n\t/**\n\t * Handler for the 'touchstart' event, enables support for\n\t * swipe and pinch gestures.\n\t *\n\t * @param {object} event\n\t */\n\tonTouchStart( event ) {\n\n\t\tif( this.isSwipePrevented( event.target ) ) return true;\n\n\t\tthis.touchStartX = event.touches[0].clientX;\n\t\tthis.touchStartY = event.touches[0].clientY;\n\t\tthis.touchStartCount = event.touches.length;\n\n\t}\n\n\t/**\n\t * Handler for the 'touchmove' event.\n\t *\n\t * @param {object} event\n\t */\n\tonTouchMove( event ) {\n\n\t\tif( this.isSwipePrevented( event.target ) ) return true;\n\n\t\tlet config = this.Reveal.getConfig();\n\n\t\t// Each touch should only trigger one action\n\t\tif( !this.touchCaptured ) {\n\t\t\tthis.Reveal.onUserInput( event );\n\n\t\t\tlet currentX = event.touches[0].clientX;\n\t\t\tlet currentY = event.touches[0].clientY;\n\n\t\t\t// There was only one touch point, look for a swipe\n\t\t\tif( event.touches.length === 1 && this.touchStartCount !== 2 ) {\n\n\t\t\t\tlet availableRoutes = this.Reveal.availableRoutes({ includeFragments: true });\n\n\t\t\t\tlet deltaX = currentX - this.touchStartX,\n\t\t\t\t\tdeltaY = currentY - this.touchStartY;\n\n\t\t\t\tif( deltaX > SWIPE_THRESHOLD && Math.abs( deltaX ) > Math.abs( deltaY ) ) {\n\t\t\t\t\tthis.touchCaptured = true;\n\t\t\t\t\tif( config.navigationMode === 'linear' ) {\n\t\t\t\t\t\tif( config.rtl ) {\n\t\t\t\t\t\t\tthis.Reveal.next();\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tthis.Reveal.prev();\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tthis.Reveal.left();\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse if( deltaX < -SWIPE_THRESHOLD && Math.abs( deltaX ) > Math.abs( deltaY ) ) {\n\t\t\t\t\tthis.touchCaptured = true;\n\t\t\t\t\tif( config.navigationMode === 'linear' ) {\n\t\t\t\t\t\tif( config.rtl ) {\n\t\t\t\t\t\t\tthis.Reveal.prev();\n\t\t\t\t\t\t}\n\t\t\t\t\t\telse {\n\t\t\t\t\t\t\tthis.Reveal.next();\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tthis.Reveal.right();\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse if( deltaY > SWIPE_THRESHOLD && availableRoutes.up ) {\n\t\t\t\t\tthis.touchCaptured = true;\n\t\t\t\t\tif( config.navigationMode === 'linear' ) {\n\t\t\t\t\t\tthis.Reveal.prev();\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tthis.Reveal.up();\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\telse if( deltaY < -SWIPE_THRESHOLD && availableRoutes.down ) {\n\t\t\t\t\tthis.touchCaptured = true;\n\t\t\t\t\tif( config.navigationMode === 'linear' ) {\n\t\t\t\t\t\tthis.Reveal.next();\n\t\t\t\t\t}\n\t\t\t\t\telse {\n\t\t\t\t\t\tthis.Reveal.down();\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\t// If we're embedded, only block touch events if they have\n\t\t\t\t// triggered an action\n\t\t\t\tif( config.embedded ) {\n\t\t\t\t\tif( this.touchCaptured || this.Reveal.isVerticalSlide() ) {\n\t\t\t\t\t\tevent.preventDefault();\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\t// Not embedded? Block them all to avoid needless tossing\n\t\t\t\t// around of the viewport in iOS\n\t\t\t\telse {\n\t\t\t\t\tevent.preventDefault();\n\t\t\t\t}\n\n\t\t\t}\n\t\t}\n\t\t// There's a bug with swiping on some Android devices unless\n\t\t// the default action is always prevented\n\t\telse if( isAndroid ) {\n\t\t\tevent.preventDefault();\n\t\t}\n\n\t}\n\n\t/**\n\t * Handler for the 'touchend' event.\n\t *\n\t * @param {object} event\n\t */\n\tonTouchEnd( event ) {\n\n\t\tthis.touchCaptured = false;\n\n\t}\n\n\t/**\n\t * Convert pointer down to touch start.\n\t *\n\t * @param {object} event\n\t */\n\tonPointerDown( event ) {\n\n\t\tif( event.pointerType === event.MSPOINTER_TYPE_TOUCH || event.pointerType === \"touch\" ) {\n\t\t\tevent.touches = [{ clientX: event.clientX, clientY: event.clientY }];\n\t\t\tthis.onTouchStart( event );\n\t\t}\n\n\t}\n\n\t/**\n\t * Convert pointer move to touch move.\n\t *\n\t * @param {object} event\n\t */\n\tonPointerMove( event ) {\n\n\t\tif( event.pointerType === event.MSPOINTER_TYPE_TOUCH || event.pointerType === \"touch\" ) {\n\t\t\tevent.touches = [{ clientX: event.clientX, clientY: event.clientY }];\n\t\t\tthis.onTouchMove( event );\n\t\t}\n\n\t}\n\n\t/**\n\t * Convert pointer up to touch end.\n\t *\n\t * @param {object} event\n\t */\n\tonPointerUp( event ) {\n\n\t\tif( event.pointerType === event.MSPOINTER_TYPE_TOUCH || event.pointerType === \"touch\" ) {\n\t\t\tevent.touches = [{ clientX: event.clientX, clientY: event.clientY }];\n\t\t\tthis.onTouchEnd( event );\n\t\t}\n\n\t}\n\n}","import { closest } from '../utils/util.js'\n\n/**\n * Manages focus when a presentation is embedded. This\n * helps us only capture keyboard from the presentation\n * a user is currently interacting with in a page where\n * multiple presentations are embedded.\n */\n\nconst STATE_FOCUS = 'focus';\nconst STATE_BLUR = 'blur';\n\nexport default class Focus {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t\tthis.onRevealPointerDown = this.onRevealPointerDown.bind( this );\n\t\tthis.onDocumentPointerDown = this.onDocumentPointerDown.bind( this );\n\n\t}\n\n\t/**\n\t * Called when the reveal.js config is updated.\n\t */\n\tconfigure( config, oldConfig ) {\n\n\t\tif( config.embedded ) {\n\t\t\tthis.blur();\n\t\t}\n\t\telse {\n\t\t\tthis.focus();\n\t\t\tthis.unbind();\n\t\t}\n\n\t}\n\n\tbind() {\n\n\t\tif( this.Reveal.getConfig().embedded ) {\n\t\t\tthis.Reveal.getRevealElement().addEventListener( 'pointerdown', this.onRevealPointerDown, false );\n\t\t}\n\n\t}\n\n\tunbind() {\n\n\t\tthis.Reveal.getRevealElement().removeEventListener( 'pointerdown', this.onRevealPointerDown, false );\n\t\tdocument.removeEventListener( 'pointerdown', this.onDocumentPointerDown, false );\n\n\t}\n\n\tfocus() {\n\n\t\tif( this.state !== STATE_FOCUS ) {\n\t\t\tthis.Reveal.getRevealElement().classList.add( 'focused' );\n\t\t\tdocument.addEventListener( 'pointerdown', this.onDocumentPointerDown, false );\n\t\t}\n\n\t\tthis.state = STATE_FOCUS;\n\n\t}\n\n\tblur() {\n\n\t\tif( this.state !== STATE_BLUR ) {\n\t\t\tthis.Reveal.getRevealElement().classList.remove( 'focused' );\n\t\t\tdocument.removeEventListener( 'pointerdown', this.onDocumentPointerDown, false );\n\t\t}\n\n\t\tthis.state = STATE_BLUR;\n\n\t}\n\n\tisFocused() {\n\n\t\treturn this.state === STATE_FOCUS;\n\n\t}\n\n\tdestroy() {\n\n\t\tthis.Reveal.getRevealElement().classList.remove( 'focused' );\n\n\t}\n\n\tonRevealPointerDown( event ) {\n\n\t\tthis.focus();\n\n\t}\n\n\tonDocumentPointerDown( event ) {\n\n\t\tlet revealElement = closest( event.target, '.reveal' );\n\t\tif( !revealElement || revealElement !== this.Reveal.getRevealElement() ) {\n\t\t\tthis.blur();\n\t\t}\n\n\t}\n\n}","/**\n * Handles the showing and \n */\nexport default class Notes {\n\n\tconstructor( Reveal ) {\n\n\t\tthis.Reveal = Reveal;\n\n\t}\n\n\trender() {\n\n\t\tthis.element = document.createElement( 'div' );\n\t\tthis.element.className = 'speaker-notes';\n\t\tthis.element.setAttribute( 'data-prevent-swipe', '' );\n\t\tthis.element.setAttribute( 'tabindex', '0' );\n\t\tthis.Reveal.getRevealElement().appendChild( this.element );\n\n\t}\n\n\t/**\n\t * Called when the reveal.js config is updated.\n\t */\n\tconfigure( config, oldConfig ) {\n\n\t\tif( config.showNotes ) {\n\t\t\tthis.element.setAttribute( 'data-layout', typeof config.showNotes === 'string' ? config.showNotes : 'inline' );\n\t\t}\n\n\t}\n\n\t/**\n\t * Pick up notes from the current slide and display them\n\t * to the viewer.\n\t *\n\t * @see {@link config.showNotes}\n\t */\n\tupdate() {\n\n\t\tif( this.Reveal.getConfig().showNotes && this.element && this.Reveal.getCurrentSlide() && !this.Reveal.print.isPrintingPDF() ) {\n\n\t\t\tthis.element.innerHTML = this.getSlideNotes() || 'No notes on this slide.';\n\n\t\t}\n\n\t}\n\n\t/**\n\t * Updates the visibility of the speaker notes sidebar that\n\t * is used to share annotated slides. The notes sidebar is\n\t * only visible if showNotes is true and there are notes on\n\t * one or more slides in the deck.\n\t */\n\tupdateVisibility() {\n\n\t\tif( this.Reveal.getConfig().showNotes && this.hasNotes() && !this.Reveal.print.isPrintingPDF() ) {\n\t\t\tthis.Reveal.getRevealElement().classList.add( 'show-notes' );\n\t\t}\n\t\telse {\n\t\t\tthis.Reveal.getRevealElement().classList.remove( 'show-notes' );\n\t\t}\n\n\t}\n\n\t/**\n\t * Checks if there are speaker notes for ANY slide in the\n\t * presentation.\n\t */\n\thasNotes() {\n\n\t\treturn this.Reveal.getSlidesElement().querySelectorAll( '[data-notes], aside.notes' ).length > 0;\n\n\t}\n\n\t/**\n\t * Checks if this presentation is running inside of the\n\t * speaker notes window.\n\t *\n\t * @return {boolean}\n\t */\n\tisSpeakerNotesWindow() {\n\n\t\treturn !!window.location.search.match( /receiver/gi );\n\n\t}\n\n\t/**\n\t * Retrieves the speaker notes from a slide. Notes can be\n\t * defined in two ways:\n\t * 1. As a data-notes attribute on the slide \n\t * 2. As an
