#DocUpdate: Figure out a way to incorporate (updated Ref #37) README.md directly into RTD docs

[shaloo]

Based on feedback received via Rola

• Currently, we have https://genpipes.readthedocs.io/en/latest/user_guide/pipeline_ref.html User guide describing different pipelines. This content is based on Pipeline readme.md files. Those files have basic text whereas RTD docs are richer in information (links, etc.). Once we have taken in all updates in RTD and pushed them into README.md, Ref #DocUpdate: Figure out a way to convert new, updated RTD pipeline reference guide to README.md #37 then figure out a way to directly use README.md into the RTD docs instead of having two copies of similar information - README.md and the RTD pipeline reference guide. This will tremendously ease the document maintenance.
• Objective: First figure out a way to consume README.md into RTD
• Once Ref #DocUpdate: Figure out a way to convert new, updated RTD pipeline reference guide to README.md #37 is addressed, use that method for all pipelines and incorporate content directly from bitbucket for README.md files respectively.

[shaloo]

This can be achieved as follows:

1. Incorporate Pipeline reference guide in RTD docs as .md (markdown) content instead of .rst (restructured text). Figure out a way to see if Sphinx can ingest both .rst and .md files as input. [Experiment done - possible] Step GCP Tutorial #1 Done.

2. For remaining pipeline reference guide sections (RNA seq light, others pending) use md instead of .rst files. [RNA Seq Light - done, remaining 4 Pending]

3. Experiment to see if .md files can be input from a different GitHub repo (GenPipes sources) directly - or at least have scripts to checkout/copy them at build time. [Done - see Makefile and confirm.py updates in #DocUpdate: Figure out a way to incorporate (updated Ref #37) README.md directly into RTD docs #38 checkins]

4. Work with hector to see if these .md files can be reused directly in genpipes source code. [Pending]

5. Once #DocUpdate - OpenDoor page under community needs work #4 is complete, remove the pipeline reference guide from sphinx c3g/genpipes RTD sources and ensure they are pulled at build time from genpipes sources directly. [Pending]

[shaloo]

There is a caveat. If we use markdown then we cannot have variable based content for automating things such as reading up version number from bitbucket tag and populating it in doc content at RTD during doc build time. That feature is supported in .rst (restructured text) format content only. Need to make a call if we really want to keep User Guide for Pipeline reference in md form. Here is what I would recommend:

• We use rst file as that is more Sphinx and RTD friendly
• If we need to keep content only in one place, then use an empty README.md file that has a link to published pipeline reference in RTD docs.

[pphector]

Today I spent some time reading up on this issue to see if there is any way K\I could address point 4 outlined below. Previously, I prepared a special branch of GenPipes called read_the_docs (here). Essentially, the idea was to integrate the .md files to the GenPipes code as proposed by Shaloo. I was looking for ways to automate this directly in GitHub or Bitbucket, however it seems that this is not possible.

The best way I could come up with is inspired by this post. Essentially, it involves cloning the RTD repo locally, and using the git feature that allows for two remote repositories. The person doing the development would be able to do a commit twice so that the same commit updates both repositories...

This is a compromise. It is definitely not automated, and it does not allow changes that are made directly to a repository (in GitHub, for example) to be reflected in Bitbucket right away, without doing a pull, merge, and commit on a local copy. However, it is the only solution I could find since GitHub and Bitbucket don't have a way of updating one another directly. I don't know if anyone has a better idea or if we should try something else like what Shaloo suggest of just adding a link to RTD in the README pages in GenPipes. We can discuss further on Friday.

This can be achieved as follows:

1. Incorporate Pipeline reference guide in RTD docs as .md (markdown) content instead of .rst (restructured text). Figure out a way to see if Sphinx can ingest both .rst and .md files as input. [Experiment done - possible] Step GCP Tutorial #1 Done.
2. For remaining pipeline reference guide sections (RNA seq light, others pending) use md instead of .rst files. [RNA Seq Light - done, remaining 4 Pending]
3. Experiment to see if .md files can be input from a different GitHub repo (GenPipes sources) directly - or at least have scripts to checkout/copy them at build time. [Done - see Makefile and confirm.py updates in #DocUpdate: Figure out a way to incorporate (updated Ref #37) README.md directly into RTD docs #38 checkins]
4. Work with hector to see if these .md files can be reused directly in genpipes source code. [Pending]
5. Once #DocUpdate - OpenDoor page under community needs work #4 is complete, remove the pipeline reference guide from sphinx c3g/genpipes RTD sources and ensure they are pulled at build time from genpipes sources directly. [Pending]

[rdali]

@ehenrion @pphector