Migrate to mkdocs

[thomasmarwitz]

Demonstrate how mkdocs-jupyter plugin can be used to execute and render Jupyter Notebooks in a similar way to MyST.

Output: https://metalearners--81.org.readthedocs.build/en/81/

TODO:

• Fix enumeration in background.md
• Docstrings can / should be adapted to mkdocs style
• Jupyter notebooks should be adapted to mkdocs style (remove cell tags), additionally, discuss whether jupyter notebooks should be built once and checked into git for fast serving of docs (better DX than slow optuna notebook).
• Wait for feature: Add option s.t. overridden members are able to "inherit" docstrings from corresponding members in parent classes  mkdocstrings/python#194 => moved to griffe extension: https://github.com/mkdocstrings/griffe-inherited-docstrings (now works with version 1.1.1) TBD: maybe not include in template!
• Look into ruff pydocstyle (replacement to pre-commit-hook docformatter)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.73%. Comparing base (a4858da) to head (1bb2a25).

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #81   +/-   ##
=======================================
  Coverage   94.73%   94.73%           
=======================================
  Files          15       15           
  Lines        1806     1806           
=======================================
  Hits         1711     1711           
  Misses         95       95           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thomasmarwitz]

One issue I discovered with math blocks (= math formulae that are not rendered inline).

For some reason KaTeX is only able to render math blocks if there is no indentation before them which is annoying when dealing with enumerated lists:

Math works, enumeration doesn't (1. and 1.):

Enumeration works, math doesn't:

I haven't looked into MathJax yet. Other solutions could be to use unordered lists or use headings instead:

[kklein]

Thanks for looking into this @thomasmarwitz !
Is it possible that the subscripts no longer need escaping?

See

from this PR compared to

from https://metalearners.readthedocs.io/en/latest/background.html#x-learner

[thomasmarwitz]

@kklein Thanks for pointing that out, looks like I completely missed this. You are right, you don't need to escape subscript anymore.

[kklein]

Aside from the math and enumeration topic mentioned above, what do you see as hurdles and next steps?
Might it make sense to test whether our readthedocs setup works fine with mkdocs?

[pavelzw]

BTW an alternative to readthedocs could be github pages in combination with https://github.com/jimporter/mike. Disadvantage would be that there is no preview for PRs. Advantage would be that there is no external configuration necessary.

[kklein]

Sounds interesting! Can we have branch-specific deployments with GitHub-pages? Given our somewhat heavy use of formulas, we rely a lot on rendered docs in PRs.

[thomasmarwitz]

I don't know whether branch based deployments can be accomplished out of the box with mike and gh-pages, that would be interesting to keep an eye on.

I just tried out to host the mkdocs documentation with readthedocs and that worked: https://metalearners--81.org.readthedocs.build/en/81/

[thomasmarwitz]

During porting the rest of the documentation I noticed:

Mkdocs does (by default) not display methods if docstring is missing. For classes that override a method but don't add a docstring, this method is not listed in the API doc:

Example of method from RLearner:

Mkdocs does only include 2 methods (with "own" docstring):

Sphinx on the other hand includes these methods because they "inherit" the docstring:

[thomasmarwitz]

@kklein pointed out this mkdocstrings feature ¹.

It adds inherited methods (w/ a docstring) to subclasses. However, it still differs slightly from the sphinx behavior:

In the case above, SLearner overwrites evaluate w/o providing a docstring in the overwritten impl. Therefore, this method doesn't appear in contrast to sphinx's way of handling this.

But perhaps this is even desired behavior? In the sense that if you change the implementation, you should also adapt the docstring?

Footnotes

1. https://mkdocstrings.github.io/python/usage/configuration/members/#inherited_members ↩

[kklein]

But perhaps this is even desired behavior? In the sense that if you change the implementation, you should also adapt the docstring?

I can totally see that one might want to ask that question. In our case we mostly described the 'contract' of a given method/function in the docstrings, i.e. what a user may provide as input and what they can expect as an output. We don't usually say much about the 'how's.

At times, changing the implementation does not change this contract.

E.g. one might think of the folllowing:

class Shape:
    contour: list[Point]
    def surface_area(self):
        """Return the surface area in square meters."""
        return numerical_integration(self.contour)
        
class Rectange(Shape)
    def surface_area(self):
        return distance(self.cotour[2], self.contour[0]) * distance(self.contour[3], self.contour[1])

In this case we used said sphinx feature in order to DRY and reduce the risk of inconsistencies due to redundancy.

[pavelzw]

Can we have branch-specific deployments with GitHub-pages? Given our somewhat heavy use of formulas, we rely a lot on rendered docs in PRs.

I don't think so, so in this case readthedocs might be more fitting.
Note though, that with mkdocs you can easily spin up a dev docs instance using pixi run docs or pixi run mkdocs serve.

[pavelzw]

how about pixi run docs for interactive docs serving and pixi run docs-build for building the HTML?

i'm also wondering whether we should add --strict to docs-build, xref https://github.com/prefix-dev/pixi/blob/e56f2e6300ba2defbd322fc13a45f224886a89c4/pixi.toml#L122

[thomasmarwitz]

Strict seems reasonable

[kklein]

Thanks for your work @thomasmarwitz ! A couple of observations on the rendered docs:

• We have quite a few diagrams with a transparent background and black or dark grey font color. Given the dark background of the mkdocs theme, these are not so easily legible, e.g. https://metalearners--81.org.readthedocs.build/en/81/motivation/#accessing-base-models.

• Some of the math isn't rendered as intended, see e.g. the X-Learner paragraph, the R-Learner paragraph or the DR-Learner paragraph.

• For some reason 'Covariates' in the glossary is treated surprisingly, see here.

• Some cross-references aren't displayed as intended yet, see e.g. the second bullet in the example on using a first MetaLearner.

• The examples no longer show the output of the code cells, e.g. here.

• The Table of Content of this Linear Regression paragraph has three colon-subparagraphs which I don't see in the content.

• Some cross-references aren't picked up, see e.g. the first usage of {term} and {ref} in the known propensity example .

• Some notes aren't picked up properly, see e.g. the {note} reference in this paragraph.

• RST Urls from docstring aren't rendered as intended, see e.g. the arxiv links in the docs of the DR-Learner class.

• I can't find the parameter and return value type hints in the API docs, e.g. in the evaluate method docs.

[thomasmarwitz]

@kklein Thanks for examining the current progress so carefully. I checked each point to see whether there is some way of solving it. I haven't implemented everything as changing all URLs, adjusting all docstrings etc. takes some time.

• We have quite a few diagrams with a transparent background and black or dark grey font color. Given the dark background of the mkdocs theme, these are not so easily legible, e.g. metalearners--81.org.readthedocs.build/en/81/motivation#accessing-base-models.

We can disable the dark and auto-mode as a temporary work-around. The site will then be always displayed in light mode.
A solution can be the neat conditional rendering of images depending on light/dark mode, this would require a dark-mode friendly version of the images, though.

• Some of the math isn't rendered as intended, see e.g. the X-Learner paragraph, the R-Learner paragraph or the DR-Learner paragraph.

Thanks for pointing that out. As it turned out, prettier adjusted the tabwidth in markdown, that's why the version on github was always broken. I found a way to override the tabwidth for markdown with 4.

• For some reason 'Covariates' in the glossary is treated surprisingly, see here.

Fixed, typo in heading.

• Some cross-references aren't displayed as intended yet, see e.g. the second bullet in the example on using a first MetaLearner.

I have to replace all sphinx-like references with mkdocs / normal links. Just not there yet.

• The examples no longer show the output of the code cells, e.g. here.

I think, this point needs some discussion.

During development, I turned off the execute option during docs build as the building of the optuna notebook (through mkdocs) takes >20 min on my mac (even by excluding optuna, the remaning notebooks need 5 min to be all converted). This leads to all notebooks being used in their current state, some may have no output, but some have e.g. optuna.

When building the documentation in CI, we could execute all notebooks beforehand to ensure everything is up to date and then build the documentation. Executing all notebooks everytime, makes the mkdocs serve rather inconvenient as I expect high iteration speeds from such a live server.

WDYT? (also @pavelzw as you are a huge fan of the serve option)

• The Table of Content of this Linear Regression paragraph has three colon-subparagraphs which I don't see in the content.

Appears to be a bug, as a work-around we can remove Headings using "``" to wrap code-like elements. If we use that feature frequently, I can open an issue.

• Some cross-references aren't picked up, see e.g. the first usage of {term} and {ref} in the known propensity example .

This is also sphinx-specific terminology that I haven't replaced completely with the mkdocs equivalent.

• Some notes aren't picked up properly, see e.g. the {note} reference in this paragraph.

Mkdocs has nice admonition rendering: https://squidfunk.github.io/mkdocs-material/reference/admonitions/#+type:note. In jupyter notebooks, the syntactic sugar to render those admonitions is not (sadly) not available. A solution I found is to refer to the "compiled" html elements directly e.g.

<!-- mkdocs note -->
<div class="admonition note">
    <p class="admonition-title">Note</p>
    <p style="margin-top: 0.6rem">The fact that we have a fixed propensity score for all observations is not true for this dataset, we just use it for illustrational purposes.</p>
</div>

This produces a note, similar to sphinx:

Again, this is not really pretty. If we find ourselves using that a lot, we can also think about opening an issue to support the more concise markdown syntax.

• RST Urls from docstring aren't rendered as intended, see e.g. the arxiv links in the docs of the DR-Learner class.

This is also sphinx-specific terminology that I haven't replaced completely with the mkdocs equivalent.

• I can't find the parameter and return value type hints in the API docs, e.g. in the evaluate method docs.

I found a setting to render the type hints:

If we add black as a dependency in the docs feature, we get a formatted signature (that looks much better imo):

For a table like display (there are also other display options) like this:

The function needs a numpy, google or sphinx docstring: https://mkdocstrings.github.io/python/usage/configuration/docstrings/#docstring_style. There is a tendency that google docstrings has more features coming.

The nice thing here is that we don't need this google style or whatever everywhere. One can sprinkle that in if e.g. a param needs some explanation as in the screenshot. In all other cases, the param names and types seem pretty solid.

[thomasmarwitz]

I'm afraid this is taking so long - that's mainly because migration all sphinx specific syntax to mkdocs (even in docstrings or jupyter notebooks) takes some time and I can only spend a certain amount of time per week on this.

@kklein I'll ping you here explicitly once I've migrated all docstrings and jupyter notebooks.