From 41ee937ab9b42acc08054ba3c569d4ab8e1d1d8d Mon Sep 17 00:00:00 2001 From: Markus Rudolph Date: Tue, 14 May 2024 13:37:02 +0200 Subject: [PATCH] Rearrange documentation and add "learn the workflow" section (#223) * Add new documentation structure * Add more about AST generation and reference resolution * Add validations * Collapse menu for better overview * Expand on code generation * Add testing code for user verification * Fill further empty sites with life --- hugo/config.toml | 5 +- hugo/content/docs/_index.md | 11 +- hugo/content/docs/getting-started.md | 122 ---------- hugo/content/docs/introduction/_index.md | 25 ++ .../features.md} | 17 +- hugo/content/docs/introduction/playground.md | 5 + hugo/content/docs/introduction/showcases.md | 5 + hugo/content/docs/learn/_index.md | 6 + .../learn/minilogo}/_index.md | 2 +- .../minilogo}/building_an_extension/icon.png | Bin .../minilogo}/building_an_extension/index.md | 0 .../installed-extension.jpg | Bin .../building_an_extension/minilogo-vsix.jpg | Bin .../minilogo-with-icon.png | Bin .../building_an_extension/vsix-install.jpg | Bin .../building_an_extension/vsix-installed.jpg | Bin .../learn/minilogo}/customizing_cli.md | 0 .../learn/minilogo}/generation.md | 0 .../learn/minilogo}/generation_in_the_web.md | 0 .../learn/minilogo}/langium_and_monaco.md | 0 .../learn/minilogo}/validation.md | 0 .../learn/minilogo}/writing_a_grammar.md | 0 hugo/content/docs/learn/workflow/_index.md | 80 ++++++ .../docs/learn/workflow/create_validations.md | 96 ++++++++ .../docs/learn/workflow/generate_ast.md | 103 ++++++++ .../learn/workflow/generate_everything.md | 56 +++++ hugo/content/docs/learn/workflow/install.md | 16 ++ .../workflow/resolve_cross_references.md | 229 ++++++++++++++++++ hugo/content/docs/learn/workflow/scaffold.md | 71 ++++++ .../docs/learn/workflow/write_grammar.md | 58 +++++ hugo/content/docs/recipes/_index.md | 9 + .../recipes}/builtin-library.md | 2 +- .../{guides => docs/recipes}/code-bundling.md | 2 +- .../{guides => docs/recipes}/formatting.md | 0 .../recipes}/multiple-languages.md | 33 ++- .../recipes}/scoping/_index.md | 2 +- .../recipes}/scoping/class-member.md | 0 .../recipes}/scoping/qualified-name.md | 0 hugo/content/docs/reference/_index.md | 24 ++ .../{ => reference}/configuration-services.md | 2 +- .../{ => reference}/document-lifecycle.md | 2 +- hugo/content/docs/reference/glossary.md | 20 ++ .../docs/{ => reference}/grammar-language.md | 5 +- .../semantic-model.md} | 2 +- hugo/content/guides/_index.md | 8 - hugo/content/playground/_index.html | 2 +- 46 files changed, 850 insertions(+), 170 deletions(-) delete mode 100644 hugo/content/docs/getting-started.md create mode 100644 hugo/content/docs/introduction/_index.md rename hugo/content/docs/{langium-overview.md => introduction/features.md} (95%) create mode 100644 hugo/content/docs/introduction/playground.md create mode 100644 hugo/content/docs/introduction/showcases.md create mode 100644 hugo/content/docs/learn/_index.md rename hugo/content/{tutorials => docs/learn/minilogo}/_index.md (97%) rename hugo/content/{tutorials => docs/learn/minilogo}/building_an_extension/icon.png (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/building_an_extension/index.md (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/building_an_extension/installed-extension.jpg (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/building_an_extension/minilogo-vsix.jpg (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/building_an_extension/minilogo-with-icon.png (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/building_an_extension/vsix-install.jpg (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/building_an_extension/vsix-installed.jpg (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/customizing_cli.md (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/generation.md (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/generation_in_the_web.md (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/langium_and_monaco.md (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/validation.md (100%) rename hugo/content/{tutorials => docs/learn/minilogo}/writing_a_grammar.md (100%) create mode 100644 hugo/content/docs/learn/workflow/_index.md create mode 100644 hugo/content/docs/learn/workflow/create_validations.md create mode 100644 hugo/content/docs/learn/workflow/generate_ast.md create mode 100644 hugo/content/docs/learn/workflow/generate_everything.md create mode 100644 hugo/content/docs/learn/workflow/install.md create mode 100644 hugo/content/docs/learn/workflow/resolve_cross_references.md create mode 100644 hugo/content/docs/learn/workflow/scaffold.md create mode 100644 hugo/content/docs/learn/workflow/write_grammar.md create mode 100644 hugo/content/docs/recipes/_index.md rename hugo/content/{guides => docs/recipes}/builtin-library.md (99%) rename hugo/content/{guides => docs/recipes}/code-bundling.md (99%) rename hugo/content/{guides => docs/recipes}/formatting.md (100%) rename hugo/content/{guides => docs/recipes}/multiple-languages.md (94%) rename hugo/content/{guides => docs/recipes}/scoping/_index.md (99%) rename hugo/content/{guides => docs/recipes}/scoping/class-member.md (100%) rename hugo/content/{guides => docs/recipes}/scoping/qualified-name.md (100%) create mode 100644 hugo/content/docs/reference/_index.md rename hugo/content/docs/{ => reference}/configuration-services.md (99%) rename hugo/content/docs/{ => reference}/document-lifecycle.md (99%) create mode 100644 hugo/content/docs/reference/glossary.md rename hugo/content/docs/{ => reference}/grammar-language.md (99%) rename hugo/content/docs/{sematic-model.md => reference/semantic-model.md} (99%) delete mode 100644 hugo/content/guides/_index.md diff --git a/hugo/config.toml b/hugo/config.toml index b54db004..b49c87f1 100644 --- a/hugo/config.toml +++ b/hugo/config.toml @@ -80,4 +80,7 @@ enableRobotsTXT = true # (Optional, default 'title') Configure how to sort file-tree menu entries. Possible options are 'title', 'linktitle', # 'date', 'publishdate', 'expirydate' or 'lastmod'. Every option can be used with a reverse modifier as well # e.g. 'title_reverse'. - #geekdocFileTreeSortBy = "title" \ No newline at end of file + #geekdocFileTreeSortBy = "title" + + geekdocCollapseSection = true + geekdocCollapseAllSections = true \ No newline at end of file diff --git a/hugo/content/docs/_index.md b/hugo/content/docs/_index.md index 9ab17e51..e8aaead5 100644 --- a/hugo/content/docs/_index.md +++ b/hugo/content/docs/_index.md @@ -1,12 +1,5 @@ --- title: "Documentation" -weight: 100 +weight: 0 --- - -Langium is an open source language engineering tool with first-class support for the Language Server Protocol, written in TypeScript and running in Node.js. - -This reference documentation provides [an overview](/docs/langium-overview), a [getting started guide](/docs/getting-started) and a deep dive into several aspects of Langium. Additional topics are covered in [the Guides section](/guides/) and step-by-step walkthroughs are available in [the tutorials section](/tutorials/). - -## Want to contribute? - -Visit the [Langium repository](https://github.com/eclipse-langium/langium) to take part in improving Langium. + \ No newline at end of file diff --git a/hugo/content/docs/getting-started.md b/hugo/content/docs/getting-started.md deleted file mode 100644 index cf53d7ee..00000000 --- a/hugo/content/docs/getting-started.md +++ /dev/null @@ -1,122 +0,0 @@ ---- -title: "Getting Started" -weight: 50 ---- - -Before diving into Langium itself, let's get your environment ready for development: - -1. You have a working [Node environment](https://nodejs.org/en/download/) with version 16 or higher. -2. Install Yeoman and the Langium extension generator. -```bash -npm i -g yo generator-langium -``` - -For our getting started example, we would also recommend you to install the latest version of [vscode](https://code.visualstudio.com/). - -## Your first example language - -To create your first working DSL, execute the yeoman generator: - -```bash -yo langium -``` - -Yeoman will prompt you with a few basic questions about your DSL: - -1. _Extension name_: Will be used as the folder name of your extension and its `package.json`. -2. _Language name_: Will be used as the name of the grammar and as a prefix for some generated files and service classes. -3. _File extensions_: A comma separated list of file extensions for your DSL. - -Afterwards, it will generate a new project and start installing all dependencies, including the `langium` framework as well as the `langium-cli` command line tool required for generating code based on your grammar definition. - -After everything has successfully finished running, open your newly created Langium project with vscode via the UI (File > Open Folder...) or execute the following command, replacing `hello-world` with your chosen project name: - -```bash -code hello-world -``` - -Press F5 or open the debug view and start the available debug configuration to launch the extension in a new _Extension Development Host_ window. Open a folder and create a file with your chosen file extension (`.hello` is the default). The `hello-world` language accepts two kinds of entities: The `person` and `Hello` entity. Here's a quick example on how to use them both: - -``` -person Alice -Hello Alice! - -person Bob -Hello Bob! -``` - -The file `src/language/hello-world.langium` in your newly created project contains your grammar. - -## Explaining the terms - -If you're already familiar with the terms used in parsing or DSL frameworks, you can skip this short excursion and go straight to the next part. However, anyone who is new to DSL development should carefully read the following primer on the terms we are using in our documentation: - -_abstract syntax tree_: A tree of elements that represents a text document. Each element is a simple JS object that combines multiple input tokens into a single object. Commonly abbreviated as _AST_. - -_document_: An abstract term to refer to a text file on your file system or an open editor document in your IDE. - -_grammar_: Defines the form of your language. In Langium, a grammar is also responsible for describing how the _AST_ is built. - -_parser_: A program that takes a _document_ as its input and computes an _abstract syntax tree_ as its output. - -_parser rule_: A parser rule describes how a certain _AST_ element is supposed to be parsed. This is done by invoking other _parser rules_ or _terminals_. - -_terminal_: A terminal is the smallest parseable part of a document. It usually represents small pieces of text like names, numbers, keywords or comments. - -_token_: A token is a substring of the _document_ that matches a certain _terminal_. It contains information about which kind of _terminal_ it represents as well as its location in the document. - - - -## Explaining the grammar - -Here's the grammar that parses the previous text snippet: - -```langium -grammar HelloWorld - -hidden terminal WS: /\s+/; -terminal ID: /[_a-zA-Z][\w]*/; - -entry Model: (persons+=Person | greetings+=Greeting)*; - -Person: - 'person' name=ID; - -Greeting: - 'Hello' person=[Person] '!'; -``` - -Let's go through this one by one: - -```langium -grammar HelloWorld -``` - -Before we tell Langium anything about our grammar contents, we first need to give it a name - in this case it's `HelloWorld`. The `langium-cli` will pick this up to prefix any generated services with this name. - -```langium -hidden terminal WS: /\s+/; -terminal ID: /[_a-zA-Z][\w]*/; -``` - -Here we define our two needed terminals for this grammar: The whitespace `WS` and identifier `ID` terminals. Terminals parse a part of our document by matching it against their regular expression. The `WS` terminal parses any whitespace characters with the regex `/\s+/`. This allows us consume whitespaces in our document. As the terminal is declared as `hidden`, the parser will parse any whitespace and discard the results. That way, we don't have to care about how many whitespaces a user uses in their document. Secondly, we define our `ID` terminal. It parses any string that starts with an underscore or letter and continues with any amount of characters that match the `\w` regex token. It will match `Alice`, `_alice`, or `_al1c3` but not `4lice` or `#alice`. Langium is using the JS regex dialect for terminal definitions. - -```langium -entry Model: (persons+=Person | greetings+=Greeting)*; -``` - -The `Model` parser rule is the `entry` point to our grammar. Parsing always starts with the `entry` rule. Here we define a repeating group of alternatives: `persons+=Person | greetings+=Greeting`. This will always try to parse either a `Person` or a `Greeting` and add it to the respective list of `persons` or `greetings` in the `Model` object. Since the alternative is wrapped in a repeating group `*`, the parser will continue until all input has been consumed. - -```langium -Person: 'person' name=ID; -``` - -The `Person` rule starts off with the `'person'` keyword. Keywords are like terminals, in the sense that they parse a part of the document. The set of keywords and terminals create the tokens that your language is able to parse. You can imagine that the `'person'` keyword here is like an indicator to tell the parser that an object of type `Person` should be parsed. After the keyword, we assign the `Person` a name by parsing an `ID`. - -```langium -Greeting: 'Hello' person=[Person] '!'; -``` - -Like the previous rule, the `Greeting` starts with a keyword. With the `person` assignment we introduce the _cross reference_, indicated by the brackets `[]`. A cross reference will allow your grammar to reference other elements that are contained in your file or workspace. By default, Langium will try to resolve this cross reference by parsing the terminal that is associated with its `name` property. In this case, we are looking for a `Person` whose `name` property matches the parsed `ID`. - -That finishes the short introduction to Langium! Feel free to play around with the grammar and use `npm run langium:generate` to regenerate the generated TypeScript files. To go further, we suggest that you continue with our [tutorials](/tutorials/). diff --git a/hugo/content/docs/introduction/_index.md b/hugo/content/docs/introduction/_index.md new file mode 100644 index 00000000..8bb8600b --- /dev/null +++ b/hugo/content/docs/introduction/_index.md @@ -0,0 +1,25 @@ +--- +title: "What is Langium?" +weight: -100 +--- +Langium is an open source language engineering tool with first-class support for the Language Server Protocol, written in TypeScript and running in Node.js. + +## Where to go from here? + +### Features + +If you need a more detailed list of Langium features, you can find them in the [features section](/docs/features). + +### Try it out + +If you want to see Langium in action, you can follow the [showcases](/showcase) or even the [playground](/playground). + +### Learn Langium + +If you are convinced by Langium and want to learn more about it, you can start with the [learn section](/docs/learn). + +### More details + +If you are looking for more details about Langium, you can find them in the [reference section](/docs/reference). + +If you are searching for a certain guide or recipe, you can find them in the [recipes section](/docs/recipes). diff --git a/hugo/content/docs/langium-overview.md b/hugo/content/docs/introduction/features.md similarity index 95% rename from hugo/content/docs/langium-overview.md rename to hugo/content/docs/introduction/features.md index baa5ad93..14c41d0e 100644 --- a/hugo/content/docs/langium-overview.md +++ b/hugo/content/docs/introduction/features.md @@ -1,8 +1,8 @@ --- -title: "Langium Overview" -weight: 0 +title: "Features" +weight: 200 +url: /docs/features --- - Designing programming languages from the ground up is hard, independent of whether your language is a "simple" domain specific language or a full-fledged general-purpose programming language. Not only do you have to keep up with the requirements of your domain experts, but you have to deal with all the technical complexity that comes with building a language, including questions such as: @@ -14,11 +14,12 @@ This is the point where Langium comes into play. Langium aims to lower the barri In this chapter, you'll get a closer look at the requirements developers usually have to implement by themselves when building a programming language: -- [Language parsing](#language-parsing) -- [Semantic models](#semantic-models) -- [Cross references and linking](#cross-references-and-linking) -- [Workspace management](#workspace-management) -- [Editing support](#editing-support) +- [Language Parsing](#language-parsing) +- [Semantic Models](#semantic-models) +- [Cross References and Linking](#cross-references-and-linking) +- [Workspace Management](#workspace-management) +- [Editing Support](#editing-support) +- [Try it out!](#try-it-out) Langium provides out-of-the-box solutions for these problems, with the ability to fine-tune every part of it to fit your domain requirements. diff --git a/hugo/content/docs/introduction/playground.md b/hugo/content/docs/introduction/playground.md new file mode 100644 index 00000000..c39edb55 --- /dev/null +++ b/hugo/content/docs/introduction/playground.md @@ -0,0 +1,5 @@ +--- +title: "Try it out!" +weight: 400 +--- + \ No newline at end of file diff --git a/hugo/content/docs/introduction/showcases.md b/hugo/content/docs/introduction/showcases.md new file mode 100644 index 00000000..5b166c72 --- /dev/null +++ b/hugo/content/docs/introduction/showcases.md @@ -0,0 +1,5 @@ +--- +title: "Showcases" +weight: 300 +--- + \ No newline at end of file diff --git a/hugo/content/docs/learn/_index.md b/hugo/content/docs/learn/_index.md new file mode 100644 index 00000000..e8707ba3 --- /dev/null +++ b/hugo/content/docs/learn/_index.md @@ -0,0 +1,6 @@ +--- +title: "Learn Langium" +weight: 0 +url: /docs/learn +--- + diff --git a/hugo/content/tutorials/_index.md b/hugo/content/docs/learn/minilogo/_index.md similarity index 97% rename from hugo/content/tutorials/_index.md rename to hugo/content/docs/learn/minilogo/_index.md index 6295b409..cff9ad83 100644 --- a/hugo/content/tutorials/_index.md +++ b/hugo/content/docs/learn/minilogo/_index.md @@ -1,5 +1,5 @@ --- -title: "Tutorials" +title: "Minilogo tutorial" weight: 200 --- diff --git a/hugo/content/tutorials/building_an_extension/icon.png b/hugo/content/docs/learn/minilogo/building_an_extension/icon.png similarity index 100% rename from hugo/content/tutorials/building_an_extension/icon.png rename to hugo/content/docs/learn/minilogo/building_an_extension/icon.png diff --git a/hugo/content/tutorials/building_an_extension/index.md b/hugo/content/docs/learn/minilogo/building_an_extension/index.md similarity index 100% rename from hugo/content/tutorials/building_an_extension/index.md rename to hugo/content/docs/learn/minilogo/building_an_extension/index.md diff --git a/hugo/content/tutorials/building_an_extension/installed-extension.jpg b/hugo/content/docs/learn/minilogo/building_an_extension/installed-extension.jpg similarity index 100% rename from hugo/content/tutorials/building_an_extension/installed-extension.jpg rename to hugo/content/docs/learn/minilogo/building_an_extension/installed-extension.jpg diff --git a/hugo/content/tutorials/building_an_extension/minilogo-vsix.jpg b/hugo/content/docs/learn/minilogo/building_an_extension/minilogo-vsix.jpg similarity index 100% rename from hugo/content/tutorials/building_an_extension/minilogo-vsix.jpg rename to hugo/content/docs/learn/minilogo/building_an_extension/minilogo-vsix.jpg diff --git a/hugo/content/tutorials/building_an_extension/minilogo-with-icon.png b/hugo/content/docs/learn/minilogo/building_an_extension/minilogo-with-icon.png similarity index 100% rename from hugo/content/tutorials/building_an_extension/minilogo-with-icon.png rename to hugo/content/docs/learn/minilogo/building_an_extension/minilogo-with-icon.png diff --git a/hugo/content/tutorials/building_an_extension/vsix-install.jpg b/hugo/content/docs/learn/minilogo/building_an_extension/vsix-install.jpg similarity index 100% rename from hugo/content/tutorials/building_an_extension/vsix-install.jpg rename to hugo/content/docs/learn/minilogo/building_an_extension/vsix-install.jpg diff --git a/hugo/content/tutorials/building_an_extension/vsix-installed.jpg b/hugo/content/docs/learn/minilogo/building_an_extension/vsix-installed.jpg similarity index 100% rename from hugo/content/tutorials/building_an_extension/vsix-installed.jpg rename to hugo/content/docs/learn/minilogo/building_an_extension/vsix-installed.jpg diff --git a/hugo/content/tutorials/customizing_cli.md b/hugo/content/docs/learn/minilogo/customizing_cli.md similarity index 100% rename from hugo/content/tutorials/customizing_cli.md rename to hugo/content/docs/learn/minilogo/customizing_cli.md diff --git a/hugo/content/tutorials/generation.md b/hugo/content/docs/learn/minilogo/generation.md similarity index 100% rename from hugo/content/tutorials/generation.md rename to hugo/content/docs/learn/minilogo/generation.md diff --git a/hugo/content/tutorials/generation_in_the_web.md b/hugo/content/docs/learn/minilogo/generation_in_the_web.md similarity index 100% rename from hugo/content/tutorials/generation_in_the_web.md rename to hugo/content/docs/learn/minilogo/generation_in_the_web.md diff --git a/hugo/content/tutorials/langium_and_monaco.md b/hugo/content/docs/learn/minilogo/langium_and_monaco.md similarity index 100% rename from hugo/content/tutorials/langium_and_monaco.md rename to hugo/content/docs/learn/minilogo/langium_and_monaco.md diff --git a/hugo/content/tutorials/validation.md b/hugo/content/docs/learn/minilogo/validation.md similarity index 100% rename from hugo/content/tutorials/validation.md rename to hugo/content/docs/learn/minilogo/validation.md diff --git a/hugo/content/tutorials/writing_a_grammar.md b/hugo/content/docs/learn/minilogo/writing_a_grammar.md similarity index 100% rename from hugo/content/tutorials/writing_a_grammar.md rename to hugo/content/docs/learn/minilogo/writing_a_grammar.md diff --git a/hugo/content/docs/learn/workflow/_index.md b/hugo/content/docs/learn/workflow/_index.md new file mode 100644 index 00000000..0b4ae27b --- /dev/null +++ b/hugo/content/docs/learn/workflow/_index.md @@ -0,0 +1,80 @@ +--- +title: "Langium's workflow" +weight: 0 +url: /docs/learn/worflow +--- + +Langium's workflow can be expressed as a flow chart diagram, which boils down to the following steps in the diagram. +Be aware of the fact that the possibilities go beyond this simple workflow. For more advanced topics, you can find answers in the [recipes](/docs/recipes). + +{{}} +flowchart TD + A(["1. Install Yeoman"]); + B(["2. Scaffold a Langium project"]); + C(["3. Write the grammar"]); + D(["4. Generate the AST"]); + E(["5. Resolve cross-references"]); + F(["6. Create validations"]); + G(["7. Generate artifacts"]); + H(["Find advanced topics"]); + A --> B --> C --> D --> E --> F --> G ~~~ H; + G -- for each additional\ngrammar change --> C; + + click A "/docs/learn/workflow/install" + click B "/docs/learn/workflow/scaffold" + click C "/docs/learn/workflow/write_grammar" + click D "/docs/learn/workflow/generate_ast" + click E "/docs/learn/workflow/resolve_cross_references" + click F "/docs/learn/workflow/create_validations" + click G "/docs/learn/workflow/generate_everything" + click H "/docs/recipes" +{{}} + +## Explanation + +This is the workflow we recommend for developing a language with Langium. It is a step-by-step guide that will help you to get started with Langium and to understand the basics of language development. + +This simple introduction can be seen as three main parts: + +* setting up your project environment (1.+2.): this is only done once +* specifying the language features (3.-7.): this cycle you need to go through for each grammar change +* everything advanced (8.): The limit of the common workflow is reached here. For specific questions you can find answers in the [recipes](/docs/recipes). + +While the first part is straight-forward, the last part is about advanced topics that differ from project to project. +The middle part will be explained briefly in the following section. + +## Initial setup + +### [1. Install Yeoman](/docs/learn/workflow/install) + +This step ensures that you start a Langium project with the Yeoman generator. Yeoman is a scaffolding tool that helps you to start a new project with a predefined structure. + +### [2. Scaffold a Langium project](/docs/learn/workflow/scaffold) + +After installing Yeoman, you can scaffold a new Langium project. + +## Core workflow + +### [3. Write the grammar](/docs/learn/workflow/write_grammar) + +The first step in the core workflow starts with the grammar. You will have some language feature in mind that you want to implement. The grammar is used to nail down the syntax of your features. You can use our Langium VS Code extension to get syntax highlighting and code completion for `.langium` files. If your grammar is free of errors, you can generate the files for the _abstract syntax tree (AST)_. + +### [4. Generate the AST](/docs/learn/workflow/generate_ast) + +The AST is the backbone of your language. It is used to represent the structure of your language elements. The AST is generated from the grammar. One important part of the AST are the _cross-references_. They are used to resolve references between language elements. If you have cross-references in your language, you need to _resolve_ them, after this step. The actual generation is done by a call of the Langium CLI. + +### [5. Resolve cross-references](/docs/learn/workflow/resolve_cross_references) + +The cross-references are used to resolve references between language elements (between different sub trees of one file or even elements of other files(!)). This step is quite important, because it is the basis for the next steps. You can also see it like this: Step 4 will generate an AST with gaps, this fifth step will fill these gaps. + +### [6. Create validations](/docs/learn/workflow/create_validations) + +From here we have a fully utilized AST. Now every input file that matches the syntax will be accepted. But we want to have more control over the input. We want to check if the input is semantically correct. This is done by creating _validations_. They are used to check the input against a set of rules. If the input does not match the rules, an error will be thrown. + +### [7. Generate artifacts](/docs/learn/workflow/generate_everything) + +Now you have a fully working language. You can generate whatever you want from the input. This can be code, documentation, or anything else. You can use the AST to traverse the input and generate the output. + +## [Find advanced topics](/docs/recipes) + +Everything that is out of the scope of the common workflow is covered in the recipes. Here you can find answers to specific questions or problems that you might encounter during the development of your language. diff --git a/hugo/content/docs/learn/workflow/create_validations.md b/hugo/content/docs/learn/workflow/create_validations.md new file mode 100644 index 00000000..5022506d --- /dev/null +++ b/hugo/content/docs/learn/workflow/create_validations.md @@ -0,0 +1,96 @@ +--- +title: "6. Create validations" +weight: 700 +url: /docs/learn/workflow/create_validations +--- + +After resolving the cross-references, you can assume that the syntax tree is complete. Now you can start with the validation of the input files. The validation process is a crucial part of the language engineering workflow. The parser ensures the syntactic correctness of the input files. The validation process ensures the semantic correctness of the input files. + +## Example + +Let's consider the Hello-World example from the Yeoman generator. One semantic of this language could be that each declared person must be greeted at most once. To be clear, the following input file is invalid, we are greeting John twice: + +```text +person John +person Jane + +Hello John! +Hello Jane! +Hello John! //should throw: You can great each person at most once! This is the 2nd greeting to John. +``` + +## Implementation + +To accomplish this, you need to implement a validator. The validator is a visitor that traverses a certain part of the syntax tree and checks for semantic errors. The following code snippet shows how you can implement a validator for the Hello-World example. Mind that the Hello-World already has a validator, you just need to add the following one. + +```ts +import type { ValidationAcceptor, ValidationChecks } from 'langium'; +import type { HelloWorldAstType, Model, Person } from './generated/ast.js'; +import type { HelloWorldServices } from './hello-world-module.js'; + +export function registerValidationChecks(services: HelloWorldServices) { + const registry = services.validation.ValidationRegistry; + const validator = services.validation.HelloWorldValidator; + const checks: ValidationChecks = { + //registers a validator for all Model AST nodes + Model: validator.checkPersonAreGreetedAtMostOnce + }; + registry.register(checks, validator); +} + +export class HelloWorldValidator { + checkPersonAreGreetedAtMostOnce(model: Model, accept: ValidationAcceptor): void { + //create a multi-counter variable using a map + const counts = new Map(); + //initialize the counter for each person to zero + model.persons.forEach(p => counts.set(p, 0)); + //iterate over all greetings and count the number of greetings for each person + model.greetings.forEach(g => { + const person = g.person.ref; + //Attention! if the linker was unsucessful, person is undefined + if(person) { + //set the new value of the counter + const newValue = counts.get(person)!+1; + counts.set(person, newValue); + //if the counter is greater than 1, create a helpful error + if(newValue > 1) { + accept('error', `You can great each person at most once! This is the ${newValue}${newValue==2?'nd':'th'} greeting to ${person.name}.`, { + node: g + }); + } + } + }); + } +} +``` + +## How to test the validator? + +To test the validator, we can simply use the `parseHelper` again. The following code snippet shows how you can test the validator: + +```ts +import { createHelloWorldServices } from "./your-project//hello-world-module.js"; +import { EmptyFileSystem } from "langium"; +import { parseHelper } from "langium/test"; +import { Model } from "../../src/language/generated/ast.js"; + +//arrange +const services = createHelloWorldServices(EmptyFileSystem); +const parse = parseHelper(services.HelloWorld); + +//act +const document = await parse(` + person John + person Jane + + Hello John! + Hello Jane! + Hello John! +`, { validation: true }); //enable validation, otherwise the validator will not be called! + +//assert +expect(document.diagnostics).toHaveLength(1); +expect(document.diagnostics![0].message).toBe('You can great each person at most once! This is the 2nd greeting to John.'); +``` + +The `expect` function can be any assertion library you like. The `Hello world` example uses Vitest. diff --git a/hugo/content/docs/learn/workflow/generate_ast.md b/hugo/content/docs/learn/workflow/generate_ast.md new file mode 100644 index 00000000..22f540bc --- /dev/null +++ b/hugo/content/docs/learn/workflow/generate_ast.md @@ -0,0 +1,103 @@ +--- +title: "4. Generate the AST" +weight: 500 +url: /docs/learn/workflow/generate_ast +--- + +After defining the grammar, you can generate the abstract syntax tree (AST) of your language. The AST is a tree representation of the source code that can be used to analyze and transform the code. The AST definition is generated by the Langium CLI. Simply call the followin command on your terminal: + +```bash +npm run langium:generate +``` + +This line will call `langium generate` on your Langium project. The Langium CLI will generate the files in the `src/generated` directory. +It will create the following files (depending on your given Langium configuration): + +* a _grammar_ file: which contains your entire grammar definition in JSON format. +* a _module_ file: which contains language-specific setup objects for the final module definition of your language. +* an _ast_ file: which contains the definition of your AST. +* several _syntax highlighting_ files: like for PrismJS, TextMate or Monarch. + +## The syntax tree + +An AST of your language is now ready to be get parsed. One important concept in Langium are _cross-references_. With them you can reference other elements in your language. For example, you can reference a variable in a function call. The AST will contain a reference to the variable. This is useful for code analysis and transformation. Technologies like ANTLR or other parser-only generators do not support this feature. For them you are forced to resolve these references in-place everytime the developer is confronted with them. + +After these generation steps, cross-references are not resolved yet. This is done in the next step. + +## Example + +Imagine you are using the Hello-World example from the Yeoman generator. For an input file like this you will get the following syntax tree from Langium during runtime: + +```text +person John +person Jane + +Hello John! +Hello Jane! +``` + +{{}} +graph TB + Model-->persons + Model-->greetings + + persons-->P1[Person] + P1 --> H1('person') + P1 --> N1[name] + N1 --> NL1('John') + + persons-->P2[Person] + P2 --> H2('person') + P2 --> N2[name] + N2 --> NL2('Jane') + + greetings-->G1[Greeting] + G1 --> KW1('hello') + G1 --> PRef1[Ref] + G1 --> EM1('!') + PRef1 --> QM1{?} + + greetings-->G2[Greeting] + G2 --> KW2('hello') + G2 --> PRef2[Ref] + G2 --> EM2('!') + PRef2 --> QM2{?} +{{}} + +Mind the gaps (question marks) for the cross-references inside the greetings. This job has to be done by the developer. Fortunately Langium provides a default implementation for cross-reference resolution. You can also implement your own resolution strategy. + +## How to test the parser? + +You can test the parser by comparing the generated AST with the expected AST. Here is an example: + +```typescript +import { createHelloWorldServices } from "./your-project//hello-world-module.js"; +import { EmptyFileSystem } from "langium"; +import { parseHelper } from "langium/test"; +import { Model } from "../../src/language/generated/ast.js"; + +//arrange +const services = createHelloWorldServices(EmptyFileSystem); +const parse = parseHelper(services.HelloWorld); + +//act +const document = await parse(` + person John + person Jane + + Hello John! + Hello Jane! +`); + +//assert +const model = document.parseResult.value; +expect(model.persons).toHaveLength(2); +expect(model.persons[0].name).toBe("John"); +expect(model.persons[1].name).toBe("Jane"); +expect(model.greetings).toHaveLength(2); +//be aware of the fact that the following checks will fail at this point, because the cross-references are not resolved yet +expect(model.greetings[0].person.ref?.name).toBe("John"); +expect(model.greetings[1].person.ref?.name).toBe("Jane"); +``` + +The `expect` function can be any assertion library you like. The `Hello world` example uses Vitest. diff --git a/hugo/content/docs/learn/workflow/generate_everything.md b/hugo/content/docs/learn/workflow/generate_everything.md new file mode 100644 index 00000000..39644153 --- /dev/null +++ b/hugo/content/docs/learn/workflow/generate_everything.md @@ -0,0 +1,56 @@ +--- +title: "7. Generate artifacts" +weight: 800 +url: /docs/learn/workflow/generate_everything +--- +The syntax was ensured. The semantics were checked. Your workspace is free of errors. Now the AST is a valid representation of your input file written in your language. It is time to generate some cool stuff! + +Depending on your domain and on your requirements there are different ways to generate artifacts from your AST. + +## How to write the generator? + +The simplest way is to generate text into a string. Let's print out every greeting from the `hello-world` example. + +```typescript +import type { Model } from '../language/generated/ast.js'; + +export function generateJavaScript(model: Model): string { + return `"use strict"; +${model.greetings + .map(greeting => `console.log('Hello, ${greeting.person.ref?.name}!');`) + .join("\n") +}`; +} +``` + +## How to test the generator? + +You can test the generator by comparing the generated text with the expected text. Here is an example. + +```typescript +import { EmptyFileSystem } from "langium"; +import { parseHelper } from "langium/test"; +import { createHelloWorldServices } from "./your-project/hello-world-module.js"; +import { Model } from "./your-project/generated/ast.js"; +import { generateJavaScript } from "./your-project/generator.js"; + +//arrange +const services = createHelloWorldServices(EmptyFileSystem); +const parse = parseHelper(services.HelloWorld); +const document = await parse(` + person Langium + Hello Langium! +`, {validation: true}); +expect(document.parseResult.lexerErrors).toHaveLength(0); +expect(document.parseResult.parserErrors).toHaveLength(0); +expect(document.diagnostics ?? []).toHaveLength(0); + +//act +const javaScript = generateJavaScript(document.parseResult.value); + +//assert +expect(javaScript).toBe(`"use strict"; +console.log('Hello, Langium!');`); +``` + +The `expect` function can be any assertion library you like. The `Hello world` example uses Vitest. diff --git a/hugo/content/docs/learn/workflow/install.md b/hugo/content/docs/learn/workflow/install.md new file mode 100644 index 00000000..cd369234 --- /dev/null +++ b/hugo/content/docs/learn/workflow/install.md @@ -0,0 +1,16 @@ +--- +title: "1. Install Yeoman" +weight: 200 +url: /docs/learn/workflow/install +--- + +Before diving into Langium itself, let's get your environment ready for development: + +1. You have a working [Node environment](https://nodejs.org/en/download/) with version 16 or higher. +2. Install Yeoman and the Langium extension generator. + +```bash +npm i -g yo generator-langium +``` + +For our getting started example, we would also recommend you to install the latest version of [vscode](https://code.visualstudio.com/). diff --git a/hugo/content/docs/learn/workflow/resolve_cross_references.md b/hugo/content/docs/learn/workflow/resolve_cross_references.md new file mode 100644 index 00000000..593c8d02 --- /dev/null +++ b/hugo/content/docs/learn/workflow/resolve_cross_references.md @@ -0,0 +1,229 @@ +--- +title: "5. Resolve cross-references" +weight: 600 +url: /docs/learn/workflow/resolve_cross_references +--- + +This step takes place after generating the AST. The AST definition was created and you are able to parse input files. But the AST is not complete yet. It contains _cross-references_ that are not resolved. Cross-references are used to reference other elements in your language. + +## Problem + +Let's illustrate the problem using the Hello-World example from the Yeoman generator: + +```text +person John +person Jane + +Hello John! +Hello Jane! +``` + +The following syntax tree is generated by the Langium parser during the runtime. Mind the gaps with the question marks. These are the missing pieces you want to fill out in this step. + +{{}} +graph TB + Model-->persons + Model-->greetings + + persons-->P1[Person] + P1 --> H1('person') + P1 --> N1[name] + N1 --> NL1('John') + + persons-->P2[Person] + P2 --> H2('person') + P2 --> N2[name] + N2 --> NL2('Jane') + + greetings-->G1[Greeting] + G1 --> KW1('hello') + G1 --> PRef1[Ref] + PRef1 -- $refText --> RT1('John') + G1 --> EM1('!') + PRef1 --> QM1{?} + + greetings-->G2[Greeting] + G2 --> KW2('hello') + G2 --> PRef2[Ref] + PRef2 -- $refText --> RT2('Jane') + G2 --> EM2('!') + PRef2 --> QM2{?} +{{}} + +You normally can achieve the cross-reference resolution by implementing a so-called scope provider and a scope computation. When setup correctly given syntax tree will change to this: + +{{}} +graph TB + Model-->persons + Model-->greetings + + persons-->P1[Person] + P1 --> H1('person') + P1 --> N1[name] + N1 --> NL1('John') + + persons-->P2[Person] + P2 --> H2('person') + P2 --> N2[name] + N2 --> NL2('Jane') + + greetings-->G1[Greeting] + G1 --> KW1('hello') + G1 --> PRef1[Ref] + PRef1 -- $refText --> RT1('John') + G1 --> EM1('!') + PRef1 -..-> P1 + + greetings-->G2[Greeting] + G2 --> KW2('hello') + G2 --> PRef2[Ref] + PRef2 -- $refText --> RT2('Jane') + G2 --> EM2('!') + PRef2 -..-> P2 +{{}} + +## Resolution of cross-references + +As already hinted, you can implement a scope provider and a scope computation. Fortunately, Langium comes with default implementations for both. But eventually as your language grows, you might want to implement your own strategy because the default is not sufficient. In the following sections the interpretation of the involved interfaces will be sketched. + +### Scope provider + +#### Terms + +The _scope provider_ is responsible for providing a _scope_ for a given cross-reference represented by the `ReferenceInfo` type. + +A _scope_ is a collection of AST nodes that are represented by the `AstNodeDescription` type. + +The _description_ is like a (string) path through the AST of a document. It can be also seen as a tuple of document URI, JSON path, name and type of the AST node. + +A _reference info_ contains the concrete AST reference (which points to nothing yet). The info also has a the parent AST node (a so-called container) of the reference and the property name under which you can find the reference under its container. In the form of this tuple (`container`, `property`, `reference`) Langium visits all cross-references using the scope provider's `getScope` method. + +```ts +export interface ScopeProvider { + getScope(context: ReferenceInfo): Scope; +} + +export interface ReferenceInfo { + reference: Reference + container: AstNode + property: string + index?: number +} + +export interface Scope { + getElement(name: string): AstNodeDescription | undefined; + getAllElements(): Stream; +} +``` + +#### Purpose + +So, what is the purpose of the scope provider? As mentioned above: it visits each cross-reference and tries to find the corresponding AST nodes over the entire workspace that can be a candidate for the cross-reference's place. It is important to understand that we do not decide here which of these nodes is the perfect match! That decision is part of the so-called linker of the Langium architecture. + +If your cross-reference's `$refText` contains the name `Jane` does not matter here. We need to provide all nodes that are possible at this position. So in the result, you would return `Jane` and `John` AST nodes - for both cross-references! + +The background for this behavior is that this mechanism can be used for two things: the cross-reference resolution and the code completion. The code completion needs to know all possible candidates for a given cross-reference. The resolution of the cross-reference is done by the linker: Given a scope for a certain cross-reference, the linker decides which of the candidates is the right one - for example the first candidate with the same name. + +### Scope computation + +The _scope computation_ is responsible for defining per document file... + +1. which AST nodes are getting exported to the global scope. These nodes will be collected by the so-called _index manager_. +2. which AST nodes (as descriptions) are available in the local scope of a certain AST node. This is meant as a cache computation for the scope provider. + +The _index manager_ is keeping in mind the global symbols of your language. It can be used by the scope provider to find the right candidates for a cross-reference. + +```ts +export interface ScopeComputation { + computeExports(document: LangiumDocument, cancelToken?: CancellationToken): Promise; + computeLocalScopes(document: LangiumDocument, cancelToken?: CancellationToken): Promise; +} +``` + +So, while the scope computation is defining what symbols are globally exported (like using the `export` keyword in Typescript), the scope provider is the place to implement the `import` of these symbols using the index manager and the semantics of your import logic. + +## Cross-reference resolution from a high-level perspective + +1. The AST gets generated by the parser for each document in the workspace. +2. The scope computation is called for each document in the workspace. All exported AST nodes are collected by the index manager. +3. The scope computation is called for each document in the workspace, again. All local scopes get computed and attached to the document. +4. The linker and the scope provider are called for each cross-reference in the workspace. The scope provider uses the index manager to find candidates for each cross-reference. The linker decides which candidate is the right one for each cross-reference. + +## Example + +For the Hello-World example, you can implement a scope provider and a scope computation like this (keep in mind that this is a alternative solution to the default implementation of Langium, which already works for most cases): + +```ts +import { ReferenceInfo, Scope, ScopeProvider, AstUtils, LangiumCoreServices, AstNodeDescriptionProvider, MapScope, EMPTY_SCOPE } from "langium"; +import { isGreeting, isModel } from "./generated/ast.js"; + +export class HelloWorldScopeProvider implements ScopeProvider { + private astNodeDescriptionProvider: AstNodeDescriptionProvider; + constructor(services: LangiumCoreServices) { + //get some helper services + this.astNodeDescriptionProvider = services.workspace.AstNodeDescriptionProvider; + } + getScope(context: ReferenceInfo): Scope { + //make sure which cross-reference you are handling right now + if(isGreeting(context.container) && context.property === 'person') { + //Success! We are handling the cross-reference of a greeting to a person! + + //get the root node of the document + const model = AstUtils.getContainerOfType(context.container, isModel)!; + //select all persons from this document + const persons = model.persons; + //transform them into node descriptions + const descriptions = persons.map(p => this.astNodeDescriptionProvider.createDescription(p, p.name)); + //create the scope + return new MapScope(descriptions); + } + return EMPTY_SCOPE; + } +} +``` + +Please make sure to override the default scope provider in your language module file like this: + +```ts +//... +export const HelloWorldModule: Module = { + //validation: ... + references: { + ScopeProvider: (services) => new HelloWorldScopeProvider(services) + } +}; +//... +``` + +## How to test the linking? + +You can test the linking by comparing the resolved references with the expected references. Here is the example from the last step. + +```ts +import { createHelloWorldServices } from "./your-project//hello-world-module.js"; +import { EmptyFileSystem } from "langium"; +import { parseHelper } from "langium/test"; +import { Model } from "../../src/language/generated/ast.js"; + +//arrange +const services = createHelloWorldServices(EmptyFileSystem); +const parse = parseHelper(services.HelloWorld); + +//act +const document = await parse(` + person John + person Jane + + Hello John! + Hello Jane! +`); + +//assert +const model = document.parseResult.value; +expect(model.persons).toHaveLength(2); +expect(model.greetings).toHaveLength(2); +expect(model.greetings[0].person.ref).toBe(model.persons[0]); +expect(model.greetings[1].person.ref).toBe(model.persons[1]); +``` + +The `expect` function can be any assertion library you like. The `Hello world` example uses Vitest. diff --git a/hugo/content/docs/learn/workflow/scaffold.md b/hugo/content/docs/learn/workflow/scaffold.md new file mode 100644 index 00000000..071b69c9 --- /dev/null +++ b/hugo/content/docs/learn/workflow/scaffold.md @@ -0,0 +1,71 @@ +--- +title: "2. Scaffold a Langium project" +weight: 300 +url: /docs/learn/workflow/scaffold +--- + +To create your first working DSL, execute the Yeoman generator: + +```bash +> yo langium +┌─────┐ ─┐ +┌───┐ │ ╶─╮ ┌─╮ ╭─╮ ╷ ╷ ╷ ┌─┬─╮ +│ ,´ │ ╭─┤ │ │ │ │ │ │ │ │ │ │ +│╱ ╰─ ╰─┘ ╵ ╵ ╰─┤ ╵ ╰─╯ ╵ ╵ ╵ +` ╶─╯ + +Welcome to Langium! This tool generates a VS Code extension with a "Hello World" language + to get started quickly. The extension name is an identifier used in the extension +marketplace or package registry. +❓ Your extension name: hello-world +The language name is used to identify your language in VS Code. Please provide a name to +be shown in the UI. CamelCase and kebab-case variants will be created and used in +different parts of the extension and language server. +❓ Your language name: Hello World +Source files of your language are identified by their file name extension. You can +specify multiple file extensions separated by commas. +❓ File extensions: .hello +Your language can be run inside of a VSCode extension. +❓ Include VSCode extension? Yes +You can add CLI to your language. +❓ Include CLI? Yes +You can run the language server in your web browser. +❓ Include Web worker? Yes +You can add the setup for language tests using Vitest. +❓ Include language tests? Yes +``` + +Yeoman will prompt you with a few basic questions about your DSL: + +1. _Extension name_: Will be used as the folder name of your extension and its `package.json`. +2. _Language name_: Will be used as the name of the grammar and as a prefix for some generated files and service classes. +3. _File extensions_: A comma separated list of file extensions for your DSL. + +The following questions are about the project parts you want to include in your project: + +* _VS Code extension_: will be used to run your language inside of a VS Code extension. +* _CLI_: will add a CLI to your language. +* _Web worker_: will add the setup for running the language server in your web browser. +* _Language tests_: will add the setup for language tests. + +Afterwards, it will generate a new project and start installing all dependencies, including the `langium` framework as well as the `langium-cli` command line tool required for generating code based on your grammar definition. + +After everything has successfully finished running, open your newly created Langium project with vscode via the UI (File > Open Folder...) or execute the following command, replacing `hello-world` with your chosen project name: + +```bash +code hello-world +``` + +## Sneak peek using the VS Code extension + +Press `F5` or open the debug view and start the available debug configuration to launch the extension in a new _Extension Development Host_ window. Open a folder and create a file with your chosen file extension (`.hello` is the default). The `hello-world` language accepts two kinds of entities: The `person` and `Hello` entity. Here's a quick example on how to use them both: + +```text +person Alice +Hello Alice! + +person Bob +Hello Bob! +``` + +The file `src/language/hello-world.langium` in your newly created project contains your grammar. diff --git a/hugo/content/docs/learn/workflow/write_grammar.md b/hugo/content/docs/learn/workflow/write_grammar.md new file mode 100644 index 00000000..8bebf921 --- /dev/null +++ b/hugo/content/docs/learn/workflow/write_grammar.md @@ -0,0 +1,58 @@ +--- +title: "3. Write the grammar" +weight: 400 +url: /docs/learn/workflow/write_grammar +--- +Your Langium project is now setup and ready to be used. The next step is to define the grammar of your language. The grammar is the most important part of your language definition. It defines the syntax of your language and how the language elements are structured. + +The grammar is defined in a `.langium` file. Make sure that you have installed the VS Code extension for Langium. This extension provides syntax highlighting and code completion for `.langium` files. Here's the grammar from the Hello-World example that was generated by the Yeoman generator: + +```langium +grammar HelloWorld + +hidden terminal WS: /\s+/; +terminal ID: /[_a-zA-Z][\w]*/; + +entry Model: (persons+=Person | greetings+=Greeting)*; + +Person: + 'person' name=ID; + +Greeting: + 'Hello' person=[Person] '!'; +``` + +Let's go through this one by one: + +```langium +grammar HelloWorld +``` + +Before we tell Langium anything about our grammar contents, we first need to give it a name - in this case it's `HelloWorld`. The `langium-cli` will pick this up to prefix any generated services with this name. + +```langium +hidden terminal WS: /\s+/; +terminal ID: /[_a-zA-Z][\w]*/; +``` + +Here we define our two needed terminals for this grammar: The whitespace `WS` and identifier `ID` terminals. Terminals parse a part of our document by matching it against their regular expression. The `WS` terminal parses any whitespace characters with the regex `/\s+/`. This allows us to consume whitespaces in our document. As the terminal is declared as `hidden`, the parser will parse any whitespace and discard the results. That way, we don't have to care about how many whitespaces a user uses in their document. Secondly, we define our `ID` terminal. It parses any string that starts with an underscore or letter and continues with any amount of characters that match the `\w` regex token. It will match `Alice`, `_alice`, or `_al1c3` but not `4lice` or `#alice`. Langium is using the JS regex dialect for terminal definitions. + +```langium +entry Model: (persons+=Person | greetings+=Greeting)*; +``` + +The `Model` parser rule is the `entry` point to our grammar. Parsing always starts with the `entry` rule. Here we define a repeating group of alternatives: `persons+=Person | greetings+=Greeting`. This will always try to parse either a `Person` or a `Greeting` and add it to the respective list of `persons` or `greetings` in the `Model` object. Since the alternative is wrapped in a repeating group `*`, the parser will continue until all input has been consumed. + +```langium +Person: 'person' name=ID; +``` + +The `Person` rule starts off with the `'person'` keyword. Keywords are like terminals, in the sense that they parse a part of the document. The set of keywords and terminals create the tokens that your language is able to parse. You can imagine that the `'person'` keyword here is like an indicator to tell the parser that an object of type `Person` should be parsed. After the keyword, we assign the `Person` a name by parsing an `ID`. + +```langium +Greeting: 'Hello' person=[Person] '!'; +``` + +Like the previous rule, the `Greeting` starts with a keyword. With the `person` assignment we introduce the _cross reference_, indicated by the brackets `[]`. A cross reference will allow your grammar to reference other elements that are contained in your file or workspace. By default, Langium will try to resolve this cross reference by parsing the terminal that is associated with its `name` property. In this case, we are looking for a `Person` whose `name` property matches the parsed `ID`. + +That finishes the short introduction to Langium! Feel free to play around with the grammar and use `npm run langium:generate` to regenerate the generated TypeScript files. To go further, we suggest that you continue with our [tutorials](/docs/learn/minilogo/). diff --git a/hugo/content/docs/recipes/_index.md b/hugo/content/docs/recipes/_index.md new file mode 100644 index 00000000..105b28bf --- /dev/null +++ b/hugo/content/docs/recipes/_index.md @@ -0,0 +1,9 @@ +--- +title: "Recipes" +weight: 400 +url: "/docs/recipes" +--- + +## Where to go from here? + +Take your time to study the recipes within the navigation on the left. They are designed to help you with common tasks and challenges you might face when working with Langium. If you have any questions or suggestions, feel free to [create an issue](https://github.com/eclipse-langium/langium/issues) or [start a discussion](https://github.com/eclipse-langium/langium/discussions) on the Github repository. diff --git a/hugo/content/guides/builtin-library.md b/hugo/content/docs/recipes/builtin-library.md similarity index 99% rename from hugo/content/guides/builtin-library.md rename to hugo/content/docs/recipes/builtin-library.md index fb749d5c..73a78bae 100644 --- a/hugo/content/guides/builtin-library.md +++ b/hugo/content/docs/recipes/builtin-library.md @@ -1,6 +1,6 @@ --- title: "Builtin Libraries" -weight: 300 +weight: 100 --- Languages usually offer their users some high-level programming features that they do not have to define themselves. diff --git a/hugo/content/guides/code-bundling.md b/hugo/content/docs/recipes/code-bundling.md similarity index 99% rename from hugo/content/guides/code-bundling.md rename to hugo/content/docs/recipes/code-bundling.md index 29089879..a7a09f27 100644 --- a/hugo/content/guides/code-bundling.md +++ b/hugo/content/docs/recipes/code-bundling.md @@ -1,6 +1,6 @@ --- title: "Code Bundling" -weight: 0 +weight: 900 --- When you first create a Langium project using the [Yeoman generator](/docs/getting-started/#your-first-example-language), it will only contain a plain TypeScript configuration, without any additional build processes. diff --git a/hugo/content/guides/formatting.md b/hugo/content/docs/recipes/formatting.md similarity index 100% rename from hugo/content/guides/formatting.md rename to hugo/content/docs/recipes/formatting.md diff --git a/hugo/content/guides/multiple-languages.md b/hugo/content/docs/recipes/multiple-languages.md similarity index 94% rename from hugo/content/guides/multiple-languages.md rename to hugo/content/docs/recipes/multiple-languages.md index e9b0ee0d..74789c7c 100644 --- a/hugo/content/guides/multiple-languages.md +++ b/hugo/content/docs/recipes/multiple-languages.md @@ -23,7 +23,6 @@ The entire change touches several files. Let's summarize what needs to be done: 1. the `package.json` needs to be adapted 2. the extension entry point file (`src/extension/main.ts`) needs to be changed slightly - ## Our scenario To keep this guide easy, I will use the [`hello-world` project](/docs/getting-started/). @@ -43,7 +42,7 @@ flowchart Implementation -->|requires| Configuration {{}} -## Let's start! +## Let's start ### Grammar @@ -73,6 +72,7 @@ hidden terminal SL_COMMENT: /\/\/[^\n\r]*/; Now, split it into three new files (let's call the entry rules units and the files we can name like `multiple-languages-(configuration|definition|implementation).langium`): Our definition grammar: + ```langium grammar MultiDefinition @@ -90,6 +90,7 @@ hidden terminal SL_COMMENT: /\/\/[^\n\r]*/; ``` Our configuration grammar (note the import): + ```langium grammar MultiConfiguration @@ -99,6 +100,7 @@ entry ConfigurationUnit: 'I' 'am' who=[Person:ID] '.'; ``` Our implementation grammar (note the import again): + ```langium grammar MultiImplementation @@ -192,13 +194,16 @@ After adding two more languages, some important classes get generated - which ne 2. You will notice a wrong import (which is ok, we renamed it in the previous steps and derived new classes by code generation). 3. Import the new generated modules instead. Replace this line: + ```ts import { MultipleLanguagesGeneratedModule, MultipleLanguagesGeneratedSharedModule } from './generated/module.js'; ``` + with the following: + ```ts import { MultiConfigurationGeneratedModule, @@ -207,7 +212,9 @@ After adding two more languages, some important classes get generated - which ne MultipleLanguagesGeneratedSharedModule } from './generated/module.js'; ``` + 4. In the function `createMultipleLanguagesServices` you will notice an error line now, because we deleted the old class name in the previous step. The code there needs to basically be tripled. But before we do this, we need to define the new output type of `createMultipleLanguagesServices`. In the end this should lead to this definition: + ```ts export function createMultipleLanguagesServices(context: DefaultSharedModuleContext): { shared: LangiumSharedServices, @@ -253,10 +260,10 @@ After this step, Langium is set up correctly. But if you try to build now, the c Let's clean up the error lines. Here are some general hints: * keep in mind, that you are dealing with three file types now, namely `*.me`, `*.who` and `*.hello` - * you can distinguish them very easily by selecting the right sub service from the result object of `createMultipleLanguagesServices`, which is either `Configuration`, `Definition` or `Implementation`, but not `shared` - * all these services have a sub service with file extensions: `[Configuration,Definition,...].LanguageMetaData.fileExtensions: string[]` - * so, when you are obtaining any documents from the `DocumentBuilder` you can be sure that they are parsed by the matching language service - * to distinguish them on your own, use the AST functions for determining the root type, for example for the Configuration language use `isConfigurationUnit(document.parseResult.value)` + * you can distinguish them very easily by selecting the right sub service from the result object of `createMultipleLanguagesServices`, which is either `Configuration`, `Definition`, or `Implementation`, but not `shared` + * all these services have a sub service with file extensions: `[Configuration,Definition,...].LanguageMetaData.fileExtensions: string[]` + * so, when you are obtaining any documents from the `DocumentBuilder` you can be sure that they are parsed by the matching language service + * to distinguish them on your own, use the AST functions for determining the root type, for example for the Configuration language use `isConfigurationUnit(document.parseResult.value)` ### VSCode extension @@ -371,7 +378,7 @@ const clientOptions: LanguageClientOptions = { }; ``` -## Test the extension! +## Test the extension Now everything should be executable. **Do not forget to build**! @@ -379,7 +386,7 @@ Let's run the extension and create some files in our workspace: ### Definition `people.who` -``` +```text person Markus person Michael person Frank @@ -387,13 +394,13 @@ person Frank ### Configuration `thats.me` -``` +```text I am Markus. ``` ### Implementation `greetings.hello` -``` +```text Hello Markus! Hello Michael! ``` @@ -401,11 +408,12 @@ Hello Michael! ## Checklist You should be able now...: + * to see proper syntax highlighting * to trigger auto completion for keywords * to jump to the definition by Cmd/Ctrl-clicking on a person's name -# Add a validator (task) +## Add a validator (task) As promised, let's add a simple validation rule, that you cannot greet yourself. Therefore we enter our name in the `thats.me` file like we did in the previous step. @@ -428,7 +436,7 @@ checkNotGreetingYourself(greeting: Greeting, accept: ValidationAcceptor): void { After doing so, your name should display a warning, stating that you cannot greet yourself. -# Troubleshooting +## Troubleshooting In this section we will list common mistakes. @@ -437,4 +445,3 @@ In this section we will list common mistakes. * Since we are basically just copy-pasting given configuration, be aware of what you are pasting. Make sure that the code still makes sense after copying. You probably forgot to adapt the pasted code. If you encounter any problems, we are happy to help in our [discussions](https://github.com/eclipse-langium/langium/discussions) page or our [issue tracker](https://github.com/eclipse-langium/langium/issues). - diff --git a/hugo/content/guides/scoping/_index.md b/hugo/content/docs/recipes/scoping/_index.md similarity index 99% rename from hugo/content/guides/scoping/_index.md rename to hugo/content/docs/recipes/scoping/_index.md index e75d086e..42afe56a 100644 --- a/hugo/content/guides/scoping/_index.md +++ b/hugo/content/docs/recipes/scoping/_index.md @@ -1,6 +1,6 @@ --- title: "Scoping" -weight: 100 +weight: 0 --- You likely know scopes from programming, where some variables are only available from certain areas (such as blocks) in your program. For example, take the short Typescript snippet below. Based on the block (scope) where a variable is declared, it may or may not be available at another location in the same program. diff --git a/hugo/content/guides/scoping/class-member.md b/hugo/content/docs/recipes/scoping/class-member.md similarity index 100% rename from hugo/content/guides/scoping/class-member.md rename to hugo/content/docs/recipes/scoping/class-member.md diff --git a/hugo/content/guides/scoping/qualified-name.md b/hugo/content/docs/recipes/scoping/qualified-name.md similarity index 100% rename from hugo/content/guides/scoping/qualified-name.md rename to hugo/content/docs/recipes/scoping/qualified-name.md diff --git a/hugo/content/docs/reference/_index.md b/hugo/content/docs/reference/_index.md new file mode 100644 index 00000000..b6ab5a27 --- /dev/null +++ b/hugo/content/docs/reference/_index.md @@ -0,0 +1,24 @@ +--- +title: "Reference" +weight: 300 +--- + +This section contains the reference documentation for Langium. + +## Where to go from here? + +### Glossary + +If you are looking for a specific term or concept, you can find it in the [glossary](/docs/reference/glossary). + +### Grammar language + +If you are looking for a specific grammar language feature, you can find it in the [grammar language](/docs/reference/grammar-language). + +### Architecture + +If you are looking for a specific architecture feature, here are some nice readings: + +* [Configuration via services](/docs/reference/configuration-services) +* [Document lifecycle](/docs/reference/document-lifecycle) +* [Semantic model](/docs/reference/semantic-model) diff --git a/hugo/content/docs/configuration-services.md b/hugo/content/docs/reference/configuration-services.md similarity index 99% rename from hugo/content/docs/configuration-services.md rename to hugo/content/docs/reference/configuration-services.md index 8dbad125..56ffb0e8 100644 --- a/hugo/content/docs/configuration-services.md +++ b/hugo/content/docs/reference/configuration-services.md @@ -1,6 +1,6 @@ --- title: "Configuration via Services" -weight: 300 +weight: 200 --- Langium supports the configuration of most aspects of your language and language server via a set of *services*. Those services are configured by *modules*, which are essentially mappings from a service name to its implementation. diff --git a/hugo/content/docs/document-lifecycle.md b/hugo/content/docs/reference/document-lifecycle.md similarity index 99% rename from hugo/content/docs/document-lifecycle.md rename to hugo/content/docs/reference/document-lifecycle.md index 3672e486..1a99770e 100644 --- a/hugo/content/docs/document-lifecycle.md +++ b/hugo/content/docs/reference/document-lifecycle.md @@ -1,6 +1,6 @@ --- title: 'Document Lifecycle' -weight: 400 +weight: 300 --- `LangiumDocument` is the central data structure in Langium that represents a text file of your DSL. Its main purpose is to hold the parsed Abstract Syntax Tree (AST) plus additional information derived from it. After its creation, a `LangiumDocument` must be "built" before it can be used in any way. The service responsible for building documents is called `DocumentBuilder`. diff --git a/hugo/content/docs/reference/glossary.md b/hugo/content/docs/reference/glossary.md new file mode 100644 index 00000000..b84a435d --- /dev/null +++ b/hugo/content/docs/reference/glossary.md @@ -0,0 +1,20 @@ +--- +title: "Glossary" +weight: 50 +--- + +Anyone who is new to DSL development should carefully read the following primer on the terms we are using in our documentation: + +_abstract syntax tree_: A tree of elements that represents a text document. Each element is a simple JS object that combines multiple input tokens into a single object. Commonly abbreviated as _AST_. + +_document_: An abstract term to refer to a text file on your file system or an open editor document in your IDE. + +_grammar_: Defines the form of your language. In Langium, a grammar is also responsible for describing how the _AST_ is built. + +_parser_: A program that takes a _document_ as its input and computes an _abstract syntax tree_ as its output. + +_parser rule_: A parser rule describes how a certain _AST_ element is supposed to be parsed. This is done by invoking other _parser rules_ or _terminals_. + +_terminal_: A terminal is the smallest parsable part of a document. It usually represents small pieces of text like names, numbers, keywords or comments. + +_token_: A token is a substring of the _document_ that matches a certain _terminal_. It contains information about which kind of _terminal_ it represents as well as its location in the document. \ No newline at end of file diff --git a/hugo/content/docs/grammar-language.md b/hugo/content/docs/reference/grammar-language.md similarity index 99% rename from hugo/content/docs/grammar-language.md rename to hugo/content/docs/reference/grammar-language.md index 54c443d5..17266f9e 100644 --- a/hugo/content/docs/grammar-language.md +++ b/hugo/content/docs/reference/grammar-language.md @@ -1,7 +1,10 @@ --- -title: "The Grammar Language" +title: "Grammar Language" weight: 100 --- + +{{< toc format=html >}} + The grammar language describes the syntax and structure of your language. The [Langium grammar language](https://github.com/eclipse-langium/langium/blob/main/packages/langium/src/grammar/langium-grammar.langium) is implemented using Langium itself and therefore follows the same syntactic rules as any language created with Langium. The grammar language will define the structure of the *abstract syntax tree* (AST) which in Langium is a collection of *TypeScript types* describing the content of a parsed document and organized hierarchically. The individual nodes of the tree are then represented with JavaScript objects at runtime. In the following, we describe the Langium syntax and document structure. diff --git a/hugo/content/docs/sematic-model.md b/hugo/content/docs/reference/semantic-model.md similarity index 99% rename from hugo/content/docs/sematic-model.md rename to hugo/content/docs/reference/semantic-model.md index de480119..56df2bca 100644 --- a/hugo/content/docs/sematic-model.md +++ b/hugo/content/docs/reference/semantic-model.md @@ -1,6 +1,6 @@ --- title: "Semantic Model Inference" -weight: 200 +weight: 400 --- When AST nodes are created during the parsing of a document, they are given a type. The language grammar dictates the shape of those types and how they might be related to each other. All types form the *semantic model* of your language. There are two ways by which Langium derives semantic model types from the grammar, by **[inference](#inferred-types)** and by **[declaration](#declared-types)**. diff --git a/hugo/content/guides/_index.md b/hugo/content/guides/_index.md deleted file mode 100644 index 90f8642b..00000000 --- a/hugo/content/guides/_index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -title: "Guides" -weight: 200 ---- - -On this page, you will find guides for specific topics that you often encounter while creating your own language. This includes more advanced help on how to use specific APIs offered by Langium. - -For a more systematic walk-through, consider starting with the [tutorials](/tutorials/). diff --git a/hugo/content/playground/_index.html b/hugo/content/playground/_index.html index d005d41b..8ea1d026 100644 --- a/hugo/content/playground/_index.html +++ b/hugo/content/playground/_index.html @@ -1,6 +1,6 @@ --- title: "Playground" -weight: 0 +weight: 400 type: playground layout: index url: "/playground"