Scoping review: Planning and task tracking

[lwjohnst86]

Repo found here: https://github.com/science-collective/scoping-review

General aim of project: Scrapping websites, resources, institutions, and publications for projects and initiatives similar to this one. To map out the space and as a scoping review. This has the added advantage that it would:

• Create a list of online platforms and databases that are already out there which can help researchers
• Create a repository/list of various tools/workflows/hints of how to improve your science in relation to open science
• Compile already existing resources into one place
• Determine how much new content we need to create vs how much we should reorganize old content

Myself and @joelostblom have been adding links and resources over the years that might be a good starting place:

• https://gitlab.com/rostools/manifesto/-/issues/8
• https://gitlab.com/rostools/manifesto/-/issues/5
• Some references here: https://gitlab.com/rostools/manifesto/-/blob/references-resources-addition/references.bib

Tasks to do (for 2022-01-17 sessions)

Assign yourself to one of these tasks that you want to do.

• (@lwjohnst86) Write a contributing guidelines document, e.g. how to add to the scoping review "data" or how to use RStudio and GitHub to "push and contribute". This contributing guideline can also eventually be repurposed as a template for other projects. See Started adding contributing documentation scoping-review#1.
• (Related to above as well) Brainstorming how multiple people can contribute "data" (e.g. links, resources, publications, etc) to repository/review.
• (@joelostblom / @hchats) Find out details of how to do scoping review and add those details to the doc/instructions.md file. In branch https://github.com/science-collective/scoping-review/
• (@hchats / others) Brainstorm how to go about searching web for similar initiatives and workflows to doing science better, as well as the search terms and search engines to use. Add these details in the "Methods" section of the doc/review.md file.
• (Everyone) Keep a log/journal of your experiences working on and collaborating in this project in the doc/reflections/YOURNAME.md file, so we can use these thoughts to add to and refine how we work together, to see what works and what could be improved.

Other tasks anyone can think of?

Tasks to do (for 2022-02-21 sessions)

Assign yourself to one of these tasks that you want to do.

• Review Started adding contributing documentation scoping-review#1 and IDEA: Way of saving and storing reference information scoping-review#2
• After reference data storage format is decided on, move @danielibsen list from Scoping review: Planning and task tracking #5 (comment) over into this format in the scoping-review repo.
• Continue working on scoping-review instructions and create a PR when ready.
• Brainstorm search strategy and protocol here: Ideas for strategy for searching web, literature, and any other resource for any given topic area scoping-review#3
• (Everyone) Keep a log/journal of your experiences working on and collaborating in this project in the doc/reflections/YOURNAME.md file, so we can use these thoughts to add to and refine how we work together, to see what works and what could be improved.

Other tasks?

Tasks for 2022-04-07

• Check out task list in the repository and work on one of those items, Tasks to do for completing project scoping-review#12
• Review the open PRs https://github.com/science-collective/scoping-review/pulls

[MarioGuCBMR]

I would love to contribute to this project!

[hchats]

@joelostblom Hi Joel. Can I help you with the task of finding out details for how to do a scoping review? That's probably the part I can help with most at the moment :)

[danielibsen]

Hi everyone. I'm contributing to this part of the project. I have meetings in the afternoon (DK time), so I'll be working on this from 9-13. As a first thing, I will go through the links from Luke and Joel and organize these into different subtopics. I think this will give us a better overview of what is out there and inform us on how to make a search strategy.

I am not sure how extensive we want to make this scoping review. But we may want to write a protocol for the search when we have an overview. If we do intend to publish this at one point, I think it would be great to publish the search protocol. We may not want to do this, as it also takes more time - just a thought. A protocol do, in any case, make the aim and process very clear.

[MarioGuCBMR]

Hi all, I will work here and there in the morning, but mostly focus from 16:30 to 19:00. I agree with Daniel regarding writing a protocol. I think that the subtopics are a great way to start, as well!

To get to a defined and clear protocol, I think we need to test several searching approaches and see their pros and cons. (easier said than done). Some of us can focus on a subtopic (it will limit the number of outcomes and makes comparing easier) and try different searching approaches and report them. In that way we can compare which searching method is more efficient and less biased. This can be very freewheeling, I think, so maybe we can suggest some methods that we use/know to track efficiently literature, especially for writing reviews.

Let me know what you think!

[hchats]

Hi @danielibsen & @MarioGuCBMR. I am finishing my PhD thesis draft today so I will jump back on at about 14:00. I've created a new branch with some initial details regarding how to do a scoping review, but I agree that a protocol is the best way forward. I found a template from Joanna Briggs Institute that we could use. I like your idea, @danielibsen, of organizing the links into sub-topics! I will check in with you later.

[danielibsen]

Links organized into subtopics

Research on reproducible science

• Identified and tested reproducibility of psychological literature: https://journals.sagepub.com/doi/full/10.1177/2515245920918872
• Do trials follow their pre-specified analysis plan: https://www.jclinepi.com/article/S0895-4356(20)30197-9/fulltext
• Openness of Digital Open Science Tools: https://zenodo.org/record/4013975#.YeUxu_7MKUl
• Analysis of online coding tutorials: https://faculty.washington.edu/ajko/papers/Kim2017CodingTutorialEvaluation.pdf
• Virtual research environment CLS: https://www.webarchive.org.uk/wayback/archive/20140615234259/http://www.jisc.ac.uk/media/documents/publications/vrelandscapereport.pdf

Reporting guidelines

• reporting guidelines for studies conducted using routinely-collected health data: http://www.record-statement.org/

Commentary papers on reproducible research

How to get started

• Path to open science: https://www.nature.com/articles/s41559-017-0160
• Improving statistical analysis: https://www.pnas.org/content/115/11/2563
• 10 simple rules for making research more robust: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005412
• On better research practices: https://pigee.wordpress.com/2016/12/31/a-commitment-to-better-research-practices-brps-in-psychological-science/

Reproducible research

• Reproducible Research in Computational Science: https://europepmc.org/backend/ptpmcrender.fcgi?accid=PMC3383002&blobtype=pdf
• Reproducibility of code: https://www.nature.com/articles/d41586-020-02462-7
• Is there a reproducible crisis: https://www.nature.com/news/polopoly_fs/1.19970!/menu/main/topColumns/topLeftColumn/pdf/533452a.pdf
• 7 steps to open science: https://psycnet.apa.org/fulltext/2019-80290-002.pdf
• Open Science and reproducible research: https://vickysteeves.gitlab.io/repro-papers/open-science-reproducible-research.html
• Open science movement: https://medium.com/@beewithablog/open-is-cancelled-da7dd6f2aaaf
• Early career researcher perspective on open and reproducible science: https://medium.com/open-knowledge-in-he/making-research-open-and-reproducible-an-early-career-researchers-perspective-344fb619838d
• Sharing reproducible research: https://www.software.ac.uk/blog/2018-05-22-sharing-reproducible-research-minimum-requirements-and-desirable-features
  Sharing reproducible research: https://www.software.ac.uk/blog/2018-05-22-sharing-reproducible-research-minimum-requirements-and-desirable-features
• Terminology in reproducible research: https://arxiv.org/abs/1802.03311
• Practices in computational biology. Data management, software, collaboration, project organization, keeping track of changes and writing manuscripts: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005510

Software

• Software as an important output of scholarly research: https://arxiv.org/abs/2011.07571
• Design of R packages: https://cran.r-project.org/web/packages/tidyverse/vignettes/manifesto.html

Open data

• Why open data is important: https://molecularbrain.biomedcentral.com/articles/10.1186/s13041-020-0552-2
• Datasheets for datasets: https://arxiv.org/abs/1803.09010
• Open publishing: https://science.thewire.in/the-sciences/scholarly-publishing-infrastructure-reliability-affordability-functionality/
• Data reuse: https://peerj.com/articles/175/
• • Data publishing: https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-12-S15-S2

Workflows

• R workflows: https://mdneuzerling.com/post/data-science-workflows/

Open collaboration

• 10 simple rules for establishing international research collaboration: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004311
• Creating high-performance teams: https://esajournals.onlinelibrary.wiley.com/doi/full/10.1890/130001?identityKey=7b1fdb0f-b092-4ce4-8fe6-14c943f803f6&regionCode=DE-BE&wol1URL=%2Fdoi%2F10.1890%2F130001%2Ffull
• Virtual research environments: https://www.jstage.jst.go.jp/article/dsj/12/0/12_GRDI-013/_article/-char/ja/

Tools

General

• Software publishing: https://blog.datadryad.org/2021/02/08/doing-it-right-a-better-approach-for-software-amp-data/
• Tier 4.0 protocol. Specifies the contents and organization of reproduction documentation for a project involving computations with statistical data: https://www.projecttier.org/tier-protocol/
• Checklist for open science papers: https://thewinnower.com/papers/3706-a-manuscript-checklist-for-improving-science

Collection of tools

• List of tools from Open Science MOOC: https://opensciencemooc.eu/modules/reproducible-research-and-data-analysis/
• A collection of tools: https://kbroman.org/Tools4RR/pages/resources.html

R

• R Markdown: https://bookdown.org/yihui/rmarkdown/
• Git and GitHub: https://happygitwithr.com/
• Reproducible data management and analysis using R in NOWAC: https://www.biorxiv.org/content/10.1101/644625v1
• Conventions for R Modeling Packages: https://tidymodels.github.io/model-implementation-principles/index.html
• Workflow r package: https://f1000research.com/articles/8-1749/v1
• R package renv for workflows: https://rstudio.github.io/renv/articles/renv.html
• R projects on workflows: https://github.com/jdblischak/r-project-workflows
• R package rrtools for making reproducible research: https://github.com/benmarwick/rrtools
• R Project management package: https://github.com/NikKrieger/projects

Other tools

• Cookiecutter pyopensci: https://cookiecutter-pyopensci.readthedocs.io/en/latest/
• Open endorsement: https://plaudit.pub/
• Overleaf template: https://www.overleaf.com/latex/templates/arxiv-slash-biorxiv-template/phncddwqtxpc
• OSF preprints: https://osf.io/preprints/
• Docker: https://arxiv.org/pdf/1410.0846.pdf

Similar initiatives

• Openscapes: https://www.openscapes.org/approach/
• FORRT: https://forrt.org/
• Open Science MOOC: https://opensciencemooc.eu/
• The Turing way: https://github.com/alan-turing-institute/the-turing-way
• EuSprig. State of the art in Spreadsheet Risk Management: http://www.eusprig.org/about.htm
• 101 research innovations: https://101innovations.wordpress.com/about-1/
• Open publishing: https://plos.org/open-access-week/
• Registered reports: https://lets-fund.org/better-science/#h.ts7jii2xofze
• Foster open science: https://www.fosteropenscience.eu/

Courses

• Special Topics In Data Science: Reproducibility for Data Science: https://canvas.uw.edu/courses/1354201
• Reproducible analysis in R: https://nceas.github.io/sasap-training/materials/

[danielibsen]

There is so much to focus on within this topic. I like the overview from https://opensciencemooc.eu/, although I think some of the topics can be put together. These overall pillars as I see it:

• Open science principles and open collaboration
• Reproducible Research and data analysis
• Open data
• Open software and software design
• Open access
• Open peer review
• Open outreach

Within each of these there are a range of initiatives, research papers and tools that can help towards more open science in general. @MarioGuCBMR maybe we could devise a search strategy for each of these pillars. A big thing will be to limit the number of papers to search, because that will be a huge workload and take a lot of time to screen.

I also like the research process outlined in: https://vickysteeves.gitlab.io/repro-papers/open-science-reproducible-research.html.

Search and discover -> develop idea -> design study -> acquire materials and collect data -> store data -> analyze data -> interpret findings and write report -> publish report -> communicate results

I added the "communicate results" at the end. I think this is also good to have in mind for where each of these pillars fit into the process of doing research.

@hchats happy writing on your thesis. I will look at your document on scoping reviews and the template when ready.

[hchats]

Just a thought: I could make an appointment with our research librarian to ask about potential databases to search!

[MarioGuCBMR]

I agree that we can start devising a strategy for each of the pillars, @danielibsen!

In this scoping review from 2014 (https://dx.doi.org/10.1186%2F1472-6963-14-271) they search in three databases: Medline, CINAHL, and EMBASE with the following keywords as query: “patient-centered care”, “client-centered care”, “framework” and “model”. Importantly, they include all publications from 1990 to 2012.

Thus, maybe we can start by:

1. Discuss databases and types of sources from @lwjohnst86 and Joel original search (and add more if needed).
2. Select year range (maybe we can select those from 1990 to 2022, order by impact factor (edit by Luke: definitely shouldn't use or even pay attention to impact factor, especially for a project like this), select the highest percentile and choose the oldest from there. Just a quick thought, I am sure there are better ways of doing it.)
3. Select keywords for every pillar. There are many ways of doings this. We can go for a broader approach (broader keywords) and then cluster the publications and sources into different pillars (stricter keywords). Another way is to argue why we have chosen each pillar beforehand and go directly with those. I think we will end up with something in between those two approaches.
4. We may need to find a way to detect outliers in the "clustering" that might be considered as their own pillar or be disregarded upon further inspection.
5. We may need to find a way to avoid reiteration.

What do you think @danielibsen? We can start by the datasets and sources and summarize how the links were obtained from @lwjohnst86. Of course, @hchats with the help of a librarian this would be optimal! In Joanna Briggs' institute they suggest that the best way to go with a scoping review is having it done by a librarian and peer-reviewed by another librarian.

[danielibsen]

@MarioGuCBMR I think your list is a good place to start the discussion next working day. Before then, we can all have a think about the items.

Circling back to the to-do for today concerning how to contribute, I think for writing the protocol we can use markdown and contribute directly here on GitHub or through R. We can also make supporting documents, e.g. with the specific search strategies for each pillar in each database in markdown. How many articles we find in each search and when should also be documented. I don't know if we need to go to the extent of using software specifically developed for conducting reviews like Covidence or Rayyan. We can also discuss this at our next working session.

@hchats I think it would be very valuable if we could get a librarian to look at the databases. We should also discuss whether or not we want to focus our search more on the bio-medical literature or want to go beyond that - I can see there is also a lot of this within natural sciences and humanities. That may also determine which databases to search in.

[joelostblom]

@hchats Yes please go ahead and do what you can/want! I can help out later if there are any gaps remaining.

[hchats]

Hi All - @lwjohnst86 @joelostblom @danielibsen @MarioGuCBMR In preparation for our co-working day on Monday, I've just booked an appointment to consult with our research librarian on databases. My plan is to prioritize the biomedical literature, and time allowing, get a sense of natural sciences/humanities databases too. I'll let you know how I go!