diff --git a/dev/1.0-specification/index.html b/dev/1.0-specification/index.html index 36942aa..676d734 100644 --- a/dev/1.0-specification/index.html +++ b/dev/1.0-specification/index.html @@ -1579,4 +1579,4 @@

Layer 5

- + diff --git a/dev/index.html b/dev/index.html index 14f4665..8913d33 100644 --- a/dev/index.html +++ b/dev/index.html @@ -10,9 +10,9 @@ open("1.0-specification.json", "w") do f JSON.print(f, norg(JSONTarget(), s), 2) end;

You can then invoke Pandoc as follow:

pandoc -f json -t markdown 1.0-specification.json -o 1.0-specification.md

Advanced usage

You can also generate an Abstract Syntax Tree (AST) that implements AbstractTrees.jl interface using norg. See also the AST module.

norg(s)

Public API

Norg.Norg — Module

Norg.jl provides a way to parse Neorg files in pure Julia.

It overloads Base.parse with custom targets. So far the only available target is HTMLTarget.

Example usage :

using Norg
-norg(HTMLTarget(), "Read {https://github.com/nvim-neorg/norg-specs}[the spec !]")
source
Norg.NORG_SEMANTICS_PATH — Constant

Path to the Norg semantics specification.

source
Norg.NORG_SPEC_PATH — Constant

Path to the Norg specification.

source
Norg.NORG_SPEC_ROOT — Constant

The root directory of the Norg specification.

source
Norg.NORG_STDLIB_PATH — Constant

Path to the Norg standard library.

source
Norg.consume_until — Method
consume_until(k, tokens, i)
-consume_until((k₁, k₂...), tokens, i)

Consume tokens until a token of kind k is encountered, or final token is reached.

source
Norg.findtargets! — Method
findtargets!(ast[, node])

Iterate over the tree to (re)build the targets attribute of the AST, listing all possible targets for magic links among direct children of node.

If node is not given, iterate over the whole AST, and empty! the targets attribute of the AST first.

source
Norg.getchildren — Method
getchildren(node, k)
-getchildren(node, k[, exclude])

Return all children and grandchildren of kind k. It can also exclude certain nodes from recursion.

source
Norg.idify — Method
idify(text)

Make some text suitable for using it as an id in a document.

source
Norg.norg — Method
norg([codegentarget, ] s)

Parse the input s to an AST. If codegentarget is included, return the result of code generation for the given target.

Examples

julia> norg("* Hello world!")
+norg(HTMLTarget(), "Read {https://github.com/nvim-neorg/norg-specs}[the spec !]")
source
Norg.NORG_SEMANTICS_PATH — Constant

Path to the Norg semantics specification.

source
Norg.NORG_SPEC_PATH — Constant

Path to the Norg specification.

source
Norg.NORG_SPEC_ROOT — Constant

The root directory of the Norg specification.

source
Norg.NORG_STDLIB_PATH — Constant

Path to the Norg standard library.

source
Norg.consume_until — Method
consume_until(k, tokens, i)
+consume_until((k₁, k₂...), tokens, i)

Consume tokens until a token of kind k is encountered, or final token is reached.

source
Norg.findtargets! — Method
findtargets!(ast[, node])

Iterate over the tree to (re)build the targets attribute of the AST, listing all possible targets for magic links among direct children of node.

If node is not given, iterate over the whole AST, and empty! the targets attribute of the AST first.

source
Norg.getchildren — Method
getchildren(node, k)
+getchildren(node, k[, exclude])

Return all children and grandchildren of kind k. It can also exclude certain nodes from recursion.

source
Norg.idify — Method
idify(text)

Make some text suitable for using it as an id in a document.

source
Norg.norg — Method
norg([codegentarget, ] s)

Parse the input s to an AST. If codegentarget is included, return the result of code generation for the given target.

Examples

julia> norg("* Hello world!")
 NorgDocument
 └─ (K"Heading1", 2, 8)
    └─ (K"ParagraphSegment", 4, 7)
@@ -21,10 +21,10 @@
       ├─ world
       └─ !
 julia> norg(HTMLTarget(), "* Hello world!")
-<div class="norg"><section id="section-h1-hello-world"><h1 id="h1-hello-world">Hello world&#33;</h1></section><section class="footnotes"><ol></ol></section></div>
source
Norg.parse_norg_timestamp — Method
parse_norg_timestamp(tokens, start, stop)

Parse a Norg timestamp to Julia DateTime. A timestamp has the following structure:

<day>,? <day-of-month> <month> <year> <time> <timezone>

Refer to the Norg specification for further explanations.

Example usage:

using Norg, AbstractTrees
+<div class="norg"><section id="section-h1-hello-world"><h1 id="h1-hello-world">Hello world&#33;</h1></section><section class="footnotes"><ol></ol></section></div>
source
Norg.parse_norg_timestamp — Method
parse_norg_timestamp(tokens, start, stop)

Parse a Norg timestamp to Julia DateTime. A timestamp has the following structure:

<day>,? <day-of-month> <month> <year> <time> <timezone>

Refer to the Norg specification for further explanations.

Example usage:

using Norg, AbstractTrees
 ast = norg"{@ Wed, 12th Jan - 20th Feb 2022}"
 node = first(collect(Leaves(ast)))
-Norg.parse_norg_timestamp(ast.tokens, node.start, node.stop)
source
Norg.textify — Method
textify(ast, node)

Return the raw text associated with a node.

source
Norg.@norg_str — Macro

Easily parse Norg string to an AST. This can be used in e.g. Pluto notebooks, because Base.show has a method for "text/html" type mime for ASTs.

Examples

julia> norg"* Norg Header 1 Example"
+Norg.parse_norg_timestamp(ast.tokens, node.start, node.stop)
source
Norg.textify — Method
textify(ast, node)

Return the raw text associated with a node.

source
Norg.@norg_str — Macro

Easily parse Norg string to an AST. This can be used in e.g. Pluto notebooks, because Base.show has a method for "text/html" type mime for ASTs.

Examples

julia> norg"* Norg Header 1 Example"
 NorgDocument
 └─ (K"Heading1", 2, 11)
    └─ (K"ParagraphSegment", 4, 10)
@@ -34,4 +34,4 @@
       ├─
       ├─ 1
       ├─
-      └─ Example
source

Inner API

The inner API is documented in the Norg internals page.

+ └─ Examplesource

Inner API

The inner API is documented in the Norg internals page.

diff --git a/dev/internals/ast/index.html b/dev/internals/ast/index.html index 320d284..18ef55f 100644 --- a/dev/internals/ast/index.html +++ b/dev/internals/ast/index.html @@ -1,2 +1,2 @@ -AST · Norg.jl
Edit on GitHub
+AST · Norg.jl
Edit on GitHub
diff --git a/dev/internals/codegen/html/index.html b/dev/internals/codegen/html/index.html index 2f43277..505a835 100644 --- a/dev/internals/codegen/html/index.html +++ b/dev/internals/codegen/html/index.html @@ -1,2 +1,2 @@ -HTMLCodegen · Norg.jl
Edit on GitHub

HTMLCodegen

Norg.Codegen.HTMLCodegen.FootnotesLevel — Type

Controls the position where footnotes are rendered. It can be within the lowest heading level i by setting HiFootnotes or at the root of the document or directly as it appears in the Norg document.

source
+HTMLCodegen · Norg.jl
Edit on GitHub

HTMLCodegen

Norg.Codegen.HTMLCodegen.FootnotesLevel — Type

Controls the position where footnotes are rendered. It can be within the lowest heading level i by setting HiFootnotes or at the root of the document or directly as it appears in the Norg document.

source
diff --git a/dev/internals/codegen/index.html b/dev/internals/codegen/index.html index 048d1d6..22af6b7 100644 --- a/dev/internals/codegen/index.html +++ b/dev/internals/codegen/index.html @@ -1,3 +1,3 @@ -Codegen · Norg.jl
Edit on GitHub
+Codegen · Norg.jl
Edit on GitHub
diff --git a/dev/internals/codegen/json/index.html b/dev/internals/codegen/json/index.html index 75eff31..d4b1fec 100644 --- a/dev/internals/codegen/json/index.html +++ b/dev/internals/codegen/json/index.html @@ -1,2 +1,2 @@ -JSONCodegen · Norg.jl
Edit on GitHub
+JSONCodegen · Norg.jl
Edit on GitHub
diff --git a/dev/internals/index.html b/dev/internals/index.html index 5a44983..56bff6a 100644 --- a/dev/internals/index.html +++ b/dev/internals/index.html @@ -1,2 +1,2 @@ -How parsing works · Norg.jl
Edit on GitHub

Norg internals

This page describes the internals of Norg.jl and how it parses .norg files.

There are three main steps for turning Norg files into HTML (since it's the only supported target for now).

  1. Tokenizing (identifying the different chunks of code)
  2. Parsing (create an Abstract Syntax Tree, AST)
  3. Code generation (turning the AST into HTML)

Notes on using Julia's dispatch system

Earlier Norg.jl would rely on Julia's type system, but that made the code type-unstable. That's why I refactored it using a kind of enumeration to label each token and node of the AST. I did not invent anything here, it comes straight from JuliaSyntax.jl super cool ideas. See Norg.Kinds.

That does not mean that the native type system is not used. Rather, there are some dispatch functions that are basically big if statements that dispatch a given Norg.Kinds.Kind towards what's called a strategy in Norg.jl. Then the native dispatch system can take the hand and dispatch towards the right method. This allows making the code type-stable for the compiler and improves greatly performances.

Tokenization

The first step for parsing Norg documents is transforming the input string into a more friendly array of token. A Norg.Tokens.Token labels a chunk of text according to the significance the parser could give it. For example there are token kinds for line endings, words, whitespaces, the * character... Note that a token can span several characters. This is the case for words, but also for whitespaces.

Parsing

The second step consist in turning the array of tokens into an Abstract Syntax Tree (AST). To do so, Norg.jl relies on two functions: Norg.Match.match_norg and Norg.Parser.parse_norg. The role of the former is to match a sequence of tokens to a parsing strategy, while the latter does the actual parsing. For example, while parsing a paragraph, if Norg.Match.match_norg encounters two tokens of kind LineEnding, then it returns a closing match of kind Paragraph. It is then up to parse_norg to properly close the current paragraph.

Code generation

The third and last step consist in generating code output from the AST. Norg.jl defines several code generation targets in the Norg.Codegen module. Code generation is fairly simple, as it simply walks the AST and turn each node into the correct output node. For example, a level one heading would be turned into an HTML <h1> node when generating code for the HTMLTarget target.

+How parsing works · Norg.jl
Edit on GitHub

Norg internals

This page describes the internals of Norg.jl and how it parses .norg files.

There are three main steps for turning Norg files into HTML (since it's the only supported target for now).

  1. Tokenizing (identifying the different chunks of code)
  2. Parsing (create an Abstract Syntax Tree, AST)
  3. Code generation (turning the AST into HTML)

Notes on using Julia's dispatch system

Earlier Norg.jl would rely on Julia's type system, but that made the code type-unstable. That's why I refactored it using a kind of enumeration to label each token and node of the AST. I did not invent anything here, it comes straight from JuliaSyntax.jl super cool ideas. See Norg.Kinds.

That does not mean that the native type system is not used. Rather, there are some dispatch functions that are basically big if statements that dispatch a given Norg.Kinds.Kind towards what's called a strategy in Norg.jl. Then the native dispatch system can take the hand and dispatch towards the right method. This allows making the code type-stable for the compiler and improves greatly performances.

Tokenization

The first step for parsing Norg documents is transforming the input string into a more friendly array of token. A Norg.Tokens.Token labels a chunk of text according to the significance the parser could give it. For example there are token kinds for line endings, words, whitespaces, the * character... Note that a token can span several characters. This is the case for words, but also for whitespaces.

Parsing

The second step consist in turning the array of tokens into an Abstract Syntax Tree (AST). To do so, Norg.jl relies on two functions: Norg.Match.match_norg and Norg.Parser.parse_norg. The role of the former is to match a sequence of tokens to a parsing strategy, while the latter does the actual parsing. For example, while parsing a paragraph, if Norg.Match.match_norg encounters two tokens of kind LineEnding, then it returns a closing match of kind Paragraph. It is then up to parse_norg to properly close the current paragraph.

Code generation

The third and last step consist in generating code output from the AST. Norg.jl defines several code generation targets in the Norg.Codegen module. Code generation is fairly simple, as it simply walks the AST and turn each node into the correct output node. For example, a level one heading would be turned into an HTML <h1> node when generating code for the HTMLTarget target.

diff --git a/dev/internals/kinds/index.html b/dev/internals/kinds/index.html index c5218b9..a832130 100644 --- a/dev/internals/kinds/index.html +++ b/dev/internals/kinds/index.html @@ -1,3 +1,3 @@ -Kinds · Norg.jl
Edit on GitHub

Kinds

Norg.Kinds — Module

To provide a type-stable parser, we handle types ourselves. This is directly inspired by JuliaSyntax.jl. See here

You can see kinds as an advanced Enum type that allows some kind of grouping of values. We use Kinds.Kind to classify the different types of tokens ast nodes.

source
Norg.Kinds.Kind — Type
Kind(name)
-K"name"

This is type tag, used to specify the type of tokens and AST nodes.

source
Norg.Kinds.@K_str — Macro
K"s"

The kind of a token or AST internal node with string "s". For example

  • K">" is the kind of the greater than sign token
source
+Kinds · Norg.jl
Edit on GitHub

Kinds

Norg.Kinds — Module

To provide a type-stable parser, we handle types ourselves. This is directly inspired by JuliaSyntax.jl. See here

You can see kinds as an advanced Enum type that allows some kind of grouping of values. We use Kinds.Kind to classify the different types of tokens ast nodes.

source
Norg.Kinds.Kind — Type
Kind(name)
+K"name"

This is type tag, used to specify the type of tokens and AST nodes.

source
Norg.Kinds.@K_str — Macro
K"s"

The kind of a token or AST internal node with string "s". For example

  • K">" is the kind of the greater than sign token
source
diff --git a/dev/internals/match/index.html b/dev/internals/match/index.html index 2f2c8af..7bad677 100644 --- a/dev/internals/match/index.html +++ b/dev/internals/match/index.html @@ -1,2 +1,2 @@ -Match · Norg.jl
Edit on GitHub

Match

Norg.Match.MatchResult — Type

Holds results of Match.match_norg. It has a Kinds.kind, that can be found, can be closing (i.e. closing an attached modifier), continued (as in "ignore this token and continue parsing"). Whether the parser should consume or not the current token is given by the consume field.

source
+Match · Norg.jl
Edit on GitHub

Match

Norg.Match.MatchResult — Type

Holds results of Match.match_norg. It has a Kinds.kind, that can be found, can be closing (i.e. closing an attached modifier), continued (as in "ignore this token and continue parsing"). Whether the parser should consume or not the current token is given by the consume field.

source
diff --git a/dev/internals/parser/index.html b/dev/internals/parser/index.html index 5f8135f..0b97d2a 100644 --- a/dev/internals/parser/index.html +++ b/dev/internals/parser/index.html @@ -1,2 +1,2 @@ -Parser · Norg.jl
Edit on GitHub

Parser

Norg.Parser.parse_norg — Function
parse_norg(strategy, tokens, i)

Try to parse the tokens sequence starting at index i using a given strategy.

source
+Parser · Norg.jl
Edit on GitHub

Parser

Norg.Parser.parse_norg — Function
parse_norg(strategy, tokens, i)

Try to parse the tokens sequence starting at index i using a given strategy.

source
diff --git a/dev/internals/scanners/index.html b/dev/internals/scanners/index.html index 6626280..31f8c10 100644 --- a/dev/internals/scanners/index.html +++ b/dev/internals/scanners/index.html @@ -1,2 +1,2 @@ -Scanners · Norg.jl
Edit on GitHub

Scanners

Norg.Scanners — Module

Provides the scanners for the tokenizer.

The role of a scanner is to recognize a sequence of characters and to produce a ScanResult.

source
Norg.Scanners.scan — Function
scan(pattern, input)

Scan the given input for the given pattern.

It will produce a Scanners.ScanResult.

If pattern is given, then try to fit the given patter at the start of input. If pattern is :

  • a ScanStrategy subtype : scan with the given strategy (e.g. Word or Whitespace)
  • a Kind : parse for the given kind.
  • an AbstractString : input must startswith pattern.
  • an AbstractArray : call scan on each element of pattern until one matches.
  • a Set{Char} : first character must be included in pattern.
source
+Scanners · Norg.jl
Edit on GitHub

Scanners

Norg.Scanners — Module

Provides the scanners for the tokenizer.

The role of a scanner is to recognize a sequence of characters and to produce a ScanResult.

source
Norg.Scanners.scan — Function
scan(pattern, input)

Scan the given input for the given pattern.

It will produce a Scanners.ScanResult.

If pattern is given, then try to fit the given patter at the start of input. If pattern is :

  • a ScanStrategy subtype : scan with the given strategy (e.g. Word or Whitespace)
  • a Kind : parse for the given kind.
  • an AbstractString : input must startswith pattern.
  • an AbstractArray : call scan on each element of pattern until one matches.
  • a Set{Char} : first character must be included in pattern.
source
diff --git a/dev/internals/tokens/index.html b/dev/internals/tokens/index.html index 4e9c30e..ea46b6c 100644 --- a/dev/internals/tokens/index.html +++ b/dev/internals/tokens/index.html @@ -1,2 +1,2 @@ -Tokens · Norg.jl
Edit on GitHub

Tokens

Tokens

Norg.Tokens.Token — Method
 Token(kind, line, char, value)

Create a Token of kind kind with value value at line and char number char.

source
Norg.Tokens.char — Method
char(x)

Return the character number in the line corresponding to position or token x.

source

Tokenize

+Tokens · Norg.jl
Edit on GitHub

Tokens

Tokens

Norg.Tokens.Token — Method
 Token(kind, line, char, value)

Create a Token of kind kind with value value at line and char number char.

source
Norg.Tokens.char — Method
char(x)

Return the character number in the line corresponding to position or token x.

source

Tokenize

diff --git a/dev/search/index.html b/dev/search/index.html index d704741..258f7b1 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -1,2 +1,2 @@ -Search · Norg.jl

Loading search...

    +Search · Norg.jl

    Loading search...