init support for doc-comments (#91)

* init: doc comment support

Co-authored-by: Shahar Dawn Or <mightyiampresence@gmail.com>

* typos and minor fixup

* add more tests

* add note & example about definition list

* Apply some suggestions

Co-authored-by: Silvan Mosberger <github@infinisil.com>

---------

Co-authored-by: Shahar Dawn Or <mightyiampresence@gmail.com>
Co-authored-by: Silvan Mosberger <github@infinisil.com>

CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## Version 3.0.0
+
+### New Features
+
+- **Official Doc-Comments Support:** We've introduced support for official doc-comments as defined in [RFC145](https://github.com/NixOS/rfcs/pull/145). This enhancement aligns nixdoc with our latest documentation standard.
+
+### Deprecated Features
+
+- **Legacy Custom Format:** The custom nixdoc format is now considered a legacy feature. We plan to phase it out in future versions to streamline documentation practices.
+- We encourage users to transition to the official doc-comment format introduced in this release.
+- For now we will continue to maintain the legacy format, but will not accept new features or enhancements for it. This decision allows for a period of transition to the new documentation practices.
+
+See [Migration guide](./doc/migration.md) for smooth transition
+
+  by @hsjobeki; co-authored by @mightyiam
+
+  in https://github.com/nix-community/nixdoc/pull/91.
+
 ## 2.7.0
 
 - Added support to customise the attribute set prefix, which was previously hardcoded to `lib`.

README.md

@@ -10,19 +10,78 @@ function set.
 
 ## Comment format
 
-Currently, identifiers are included in the documentation if they have
-a preceding comment in multiline syntax `/* something */`.
+This tool implements a subset of the doc-comment standard specified in [RFC-145/doc-comments](https://github.com/NixOS/rfcs/blob/master/rfcs/0145-doc-strings.md).
+But, it is currently limited to generating documentation for statically analysable attribute paths only.
+In the future, it could be the role of a Nix interpreter to obtain the values to be documented and their doc-comments.
 
-Two special line beginnings are recognised:
+It is important to start doc-comments with the additional asterisk (`*`) -> `/**` which renders as a doc-comment.
+
+The content of the doc-comment should conform to the [Commonmark](https://spec.commonmark.org/0.30/) specification.
+
+### Example
+
+The following is an example of markdown documentation for new and current users of nixdoc.
+
+> Sidenote: Indentation is automatically detected and should be consistent across the content. 
+> 
+> If you are used to multiline-strings (`''`) in nix this should be intuitive to follow.
+
+````nix
+{
+  /** 
+    This function adds two numbers
+
+    # Example
+
+    ```nix
+    add 4 5
+    =>
+    9
+    ```
+
+    # Type
+
+    ```
+    add :: Number -> Number -> Number
+    ```
+
+    # Arguments
+
+    a
+    : The first number
+    
+    b
+    : The second number
+    
+  */
+  add = a: b: a + b;
+}
+````
+
+> Note: Within nixpkgs the convention of using [definition-lists](https://www.markdownguide.org/extended-syntax/#definition-lists) for documenting arguments has been established.
+
+
+## Custom nixdoc format (Legacy)
+
+You should consider migrating to the newer format described above.
+
+See [Migration guide](./doc/migration.md).
+
+### Comment format (legacy)
+
+Identifiers are included in the documentation if they have
+a preceding comment in multiline syntax `/* something */`. You should consider migrating to the new format described above.
+
+Two special line beginnings are recognized:
 
 * `Example:` Everything following this line will be assumed to be a
   verbatim usage example.
-* `Type:` This line will be interpreted as a faux type signature.
+* `Type:` This line will be interpreted as a faux-type signature.
 
 These will result in appropriate elements being inserted into the
 output.
 
-## Function arguments
+### Function arguments (legacy)
 
 Function arguments can be documented by prefixing them with a comment:
 

doc/migration.md

@@ -0,0 +1,125 @@
+# Migration Guide
+
+Upgrading from nixdoc <= 2.x.x to >= 3.0.0
+
+To leverage the new doc-comment features and prepare for the deprecation of the legacy format, follow these guidelines:
+
+## Documentation Comments
+
+- Use double asterisks `/** */` to mark comments intended as documentation. This differentiates them from internal comments and ensures they are properly processed as part of the documentation.
+
+**Example:**
+
+`lib/attrsets.nix (old format)`
+````nix
+/* Filter an attribute set by removing all attributes for which the
+   given predicate return false.
+   Example:
+     filterAttrs (n: v: n == "foo") { foo = 1; bar = 2; }
+     => { foo = 1; }
+   Type:
+     filterAttrs :: (String -> Any -> Bool) -> AttrSet -> AttrSet
+*/
+filterAttrs =
+  # Predicate taking an attribute name and an attribute value, which returns `true` to include the attribute or `false` to exclude the attribute.
+  pred:
+  # The attribute set to filter
+  set:
+  listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set));
+````
+
+->
+
+`lib/attrsets.nix (new format)`
+````nix
+/**
+  Filter an attribute set by removing all attributes for which the
+  given predicate return false.
+  
+  # Example
+
+  ```nix
+  filterAttrs (n: v: n == "foo") { foo = 1; bar = 2; }
+  => { foo = 1; }
+  ```
+
+  # Type
+  
+  ```
+  filterAttrs :: (String -> Any -> Bool) -> AttrSet -> AttrSet
+  ```
+  
+  # Arguments
+
+  **pred**
+  : Predicate taking an attribute name and an attribute value, which returns `true` to include the attribute, or `false` to exclude the attribute.
+  
+  **set**
+  : The attribute set to filter
+*/
+filterAttrs =
+  pred:
+  set:
+  listToAttrs (concatMap (name: let v = set.${name}; in if pred name v then [(nameValuePair name v)] else []) (attrNames set));
+````
+
+## Documenting Arguments
+
+With the introduction of RFC145, there is a shift in how arguments are documented. While direct "argument" documentation is not specified, you can still document arguments effectively within your doc-comments by writing explicit markdown.
+
+**Example:** Migrating **Single Argument Documentation**
+
+The approach to documenting single arguments has evolved. Instead of individual argument comments, document the function and its arguments together.
+
+> Note: Within nixpkgs the convention of using [definition-lists](https://www.markdownguide.org/extended-syntax/#definition-lists) for documenting arguments has been established.
+
+```nix
+{
+  /**
+  The `id` function returns the provided value unchanged.
+  
+  # Arguments
+  
+  `x` (Any)
+  : The value to be returned.
+  
+  */
+  id = x: x;
+}
+```
+
+If arguments require more complex documentation consider starting an extra section per argument
+
+```nix
+{
+  /**
+  The `id` function returns the provided value unchanged.
+  
+  # Arguments
+  
+  ## **x** (Any)
+  (...Some comprehensive documentation)
+
+  */
+  id = x: x;
+}
+```
+
+**Example:** Documenting Structured Arguments
+Structured arguments can be documented (described in RFC145 as 'lambda formals'), using doc-comments.
+
+```nix
+{
+  /**
+  The `add` function calculates the sum of `a` and `b`.
+  */
+  add = { 
+      /** The first number to add. */
+      a, 
+      /** The second number to add. */
+      b 
+    }: a + b;
+}
+```
+
+Ensure your documentation comments start with double asterisks to comply with the new standard. The legacy format remains supported for now but will not receive new features. It will be removed once important downstream projects have been migrated.

src/comment.rs

@@ -0,0 +1,110 @@
+use rnix::ast::{self, AstToken};
+use rnix::{match_ast, SyntaxNode};
+use rowan::ast::AstNode;
+
+/// Implements functions for doc-comments according to rfc145.
+pub trait DocComment {
+    fn doc_text(&self) -> Option<&str>;
+}
+
+impl DocComment for ast::Comment {
+    /// Function returns the contents of the doc-comment, if the [ast::Comment] is a
+    /// doc-comment, or None otherwise.
+    ///
+    /// Note: [ast::Comment] holds both the single-line and multiline comment.
+    ///
+    /// /**{content}*/
+    /// -> {content}
+    ///
+    /// It is named `doc_text` to complement [ast::Comment::text].
+    fn doc_text(&self) -> Option<&str> {
+        let text = self.syntax().text();
+        // Check whether this is a doc-comment
+        if text.starts_with(r#"/**"#) && self.text().starts_with('*') {
+            self.text().strip_prefix('*')
+        } else {
+            None
+        }
+    }
+}
+
+/// Function retrieves a doc-comment from the [ast::Expr]
+///
+/// Returns an [Option<String>] of the first suitable doc-comment.
+/// Returns [None] in case no suitable comment was found.
+///
+/// Doc-comments can appear in two places for any expression
+///
+/// ```nix
+/// # (1) directly before the expression (anonymous)
+/// /** Doc */
+/// bar: bar;
+///
+/// # (2) when assigning a name.
+/// {
+///   /** Doc */
+///   foo = bar: bar;
+/// }
+/// ```
+///
+/// If the doc-comment is not found in place (1) the search continues at place (2)
+/// More precisely before the NODE_ATTRPATH_VALUE (ast)
+/// If no doc-comment was found in place (1) or (2) this function returns None.
+pub fn get_expr_docs(expr: &SyntaxNode) -> Option<String> {
+    if let Some(doc) = get_doc_comment(expr) {
+        // Found in place (1)
+        doc.doc_text().map(|v| v.to_owned())
+    } else if let Some(ref parent) = expr.parent() {
+        match_ast! {
+            match parent {
+                ast::AttrpathValue(_) => {
+                    if let Some(doc_comment) = get_doc_comment(parent) {
+                        doc_comment.doc_text().map(|v| v.to_owned())
+                    }else{
+                        None
+                    }
+                },
+                _ => {
+                    // Yet unhandled ast-nodes
+                    None
+                }
+
+            }
+        }
+        // None
+    } else {
+        // There is no parent;
+        // No further places where a doc-comment could be.
+        None
+    }
+}
+
+/// Looks backwards from the given expression
+/// Only whitespace or non-doc-comments are allowed in between an expression and the doc-comment.
+/// Any other Node or Token stops the peek.
+fn get_doc_comment(expr: &SyntaxNode) -> Option<ast::Comment> {
+    let mut prev = expr.prev_sibling_or_token();
+    loop {
+        match prev {
+            Some(rnix::NodeOrToken::Token(ref token)) => {
+                match_ast! { match token {
+                    ast::Whitespace(_) => {
+                        prev = token.prev_sibling_or_token();
+                    },
+                    ast::Comment(it) => {
+                        if it.doc_text().is_some() {
+                            break Some(it);
+                        }else{
+                            //Ignore non-doc comments.
+                            prev = token.prev_sibling_or_token();
+                        }
+                    },
+                    _ => {
+                        break None;
+                    }
+                }}
+            }
+            _ => break None,
+        };
+    }
+}