README.Rmd

---
output: github_document
---

<!-- README.md is generated from README.Rmd. Please don't edit that file -->

```{r, include = FALSE}
knitr::opts_chunk$set(
  eval = FALSE,
  collapse = TRUE,
  comment = "#>"
)
library(DiagrammeR)
library(DiagrammeRsvg)
library(rsvg)
```

# QualMiner <a href='https://bedatadriven.github.io/QualMiner/'><img src='https://raw.githubusercontent.com/bedatadriven/QualMiner/master/media/qualminer-logo.png' align="right" width="200"/></a>

Exploring qualitative indicators via text mining methods.

This repository contains the source code. The data used in the analysis can only
be accessed by using [*ActivityInfo*](https://www.activityinfo.org/) with proper
permissions.

### Usage

The website is generated by using [bookdown](https://bookdown.org/). Here are
the common steps to generate the analysis are explained below. For more detailed
information about how bookdown works, it is advised to take a look at the
(online) book.

*Yihui Xie (2019). 
bookdown: Authoring Books and Technical Documents with R Markdown.
Chapman and Hall/CRC. ISBN 978-1138700109*
<https://bookdown.org/yihui/bookdown/>

#### 0. Authenticate

The call below authenticates the current user for the requests to get the data
from *ActivityInfo*. See [**Notes**](#notes) section for more details.

```r
activityinfo::activityInfoLogin()
```

#### 1. Retrieve data & ETL

Source `etl.R` file in the `R/` directory to pull the data from *ActivityInfo API* 
and process it to make it ready for analyses. At the end of the pull, a JSON file
containing the data will
be saved in the `data/` directory.

#### 2. Add new sections to notebook

Render [RMarkdown](https://rmarkdown.rstudio.com/) files are placed in the
`analysis/` directory.

For instance, you create a new file called, e.g.
`analysis/new-section.Rmd`.

Then, add this new file name to the `analysis/_bookdown.yml` inside the `rmd_files`
array. The place where you add the path in the array is important because the
array orders the sections of the notebook.

#### 3. Render notebook

Call this command in *R* console in order to render the bookdown site:
```r
source("R/render.R"); render_bookdown()
```

Rendering does not require any connection with the *ActivityInfo API* but the
JSON data file must exist in the `data/` directory.

You will see in the git status that there are a bunch of new files created inside
the `docs/` folder because the rendered site is set to live in the `docs/`
folder in GitHub 
([see why](https://help.github.com/en/articles/configuring-a-publishing-source-for-github-pages)).

#### 4. Publish

Once you are happy with the changes in local, commit the rendered files in the
`docs/` folder and push them to the remote. You can use the following command in
your shell:

```bash
git add docs/
git commit -m "Render site"
git push origin master
```

### Process diagram
<!-- DIAGRAM START -->
```{r graph generation, eval=TRUE, include=FALSE}
script <- 'digraph diagram {
  graph [layout=dot];
  edge [decorate=true];
  rankdir=LR;

  // AI color codes:
  // #DBF1F1 (light turquoise)
  // #242934 (almost black)
	
	// OBJECTS:
	ActivityInfoSource [shape=cylinder, 
	                    label="ActivityInfo\nSource", 
	                    style=filled color="#242934" fillcolor="#DBF1F1"];
	MainData [shape=parallelogram, label="Data generated and loaded"];
	Analyses [shape=box, label="Analyses"];
	InteractiveNotebook [shape=tab, label="Interactive\nNotebook"];
	
	// ACTIONS:
	DataPulling [shape=rect, label="Pull data\nActivityInfo API"];
	ETL [shape=rect, label="ETL\ndata preparation"];
	
	// FLOW:
  {rank=same
	  ActivityInfoSource->DataPulling;
	}
	DataPulling->ETL;
	ETL->MainData;
	MainData->Analyses;
	Analyses->InteractiveNotebook;
}'
graph <- DiagrammeR::grViz(script)
```
```{r svg to file, eval=TRUE, include=FALSE}
svg <- DiagrammeRsvg::export_svg(graph)
diagram.fname <- file.path("media", "diagram.png")
rsvg::rsvg_png(charToRaw(svg), diagram.fname)
```
```{r display, eval=TRUE, echo=FALSE, out.width = "95%"}
knitr::include_graphics(diagram.fname)
```
<!-- DIAGRAM END -->

### Technical notes

+ The data can be accessed with *ActivityInfo* API by using user credentials.
The [**ActivityInfo R Language Client**](https://github.com/bedatadriven/activityinfo-R)
provides good documentation.

+ [**dplyr**](https://cran.r-project.org/package=dplyr) package is chosen as it is
useful for rapid ad-hoc analyses. The selected analysis code can be rewritten in
base R, which is proven to be more robust and stable for production
environments, towards the end of the project.