Nebel Utility for Modular Documentation

Nebel is a Python command-line tool that implements certain routine tasks associated with creating and managing modular documentation. For example, you can use Nebel to create an instance of an assembly, procedure, concept, or reference file.

• Installing Nebel

• Setting up a content repository to use Nebel

• Naming conventions for files and directories

• Creating content with Nebel

• Adding symbolic links with Nebel

• Splitting content in one file into assemblies and modules

• Identifying orphan files

• Renaming or moving files

• Backwards-incompatible change to modular file prefixes

• Nebel versioning

• Nebel Python interpreter

Installing Nebel

The nebel utility is easy to install, as it does not require any special or non-standard Python libraries.

Prerequisites:

• Python 2.7 — installed by default on Fedora Linux.

How to install:

1. Clone this repository, as follows:

   git clone git@github.com:fbolton/nebel.git

2. Add the bin/nebel executable to your PATH. For example, in the Bash shell:

   export PATH=/path/to/your/nebel/bin:$PATH

   Ideally, you should add this export command to your ~/.bashrc file, to make Nebel permanently available.

Setting up a content repository to use Nebel

The nebel utility is designed to manage modular content in a documentation repository (presumed to be a Git repository, although that is not a requirement). Nebel imposes certain requirements on the directory structure of the content repository, as follows:

• There must be a nebel.cfg file in the top level of the working directory tree. This file is required by Nebel and nebel commands must be issued in the same directory as nebel.cfg, otherwise nebel returns an error.

• Modules (that is, procedures, concepts, and references) must be stored under a modules directory. Create a modules directory to store the module files:

  mkdir modules

• Assemblies must be stored under an assemblies directory. Create an assemblies directory to store the assembly files:

  mkdir assemblies

Naming conventions for files and directories

• Naming convention for modules

• Naming convention for assemblies

Naming convention for modules

Files under the modules directory use the following format for their file names:

modules/CATEGORY/TYPE-MODULE_ID_.adoc

CATEGORY

Provides a general way to group related modules in the context of modular documentation. A category replaces the concept of a book or guide. You can use compound categories — such as camel/enterprise-integration-patterns — which implicitly introduces more subdirectories for the sub-categories.

TYPE

A file prefix that indicates what kind of module is represented by the file. The type is one of the following values:

• proc — procedure module

• con — concept module

• ref — reference module

MODULE_ID

The unique ID for the module. The format is words separated by hyphens. In a module’s file name, MODULE_ID should be identical to the anchor ID that is defined in the module and attached to the module’s title.

For example, consider the modules/install/proc-intalling-packages.adoc file. The modules category is install and the content of this file would start with:

[id="installing-packages"]
= Installing packages

Naming convention for assemblies

Files under the assemblies directory use the following format for their file names:

assemblies/CATEGORY/assembly-MODULE_ID.adoc

CATEGORY is used in the same way as the categories for modules. In particular, closely related assemblies and modules should share the same category. For example, consider the assemblies/install/assembly-how-packages-improve-performance.adoc file. The assemblies category is install and the content of this file would start with:

[id="how-packages-improve-performance"]
= How packages improve performance

Creating content with Nebel

Nebel can create content in several ways:

• Creating modules and assemblies from a CSV file

• Creating modules from an assembly file

Creating modules and assemblies from a CSV file

The nebel utility enables you to create multiple assemblies and modules from a comma-separated values (CSV) file, which you can obtain by exporting content from a spreadsheet. This capability is designed to support a workflow where you design the high-level structure of a guide in a spreadsheet (for example, in Google sheets) and then generate the corresponding assemblies and modules from the spreadsheet data.

A typical CSV file might have a structure like the following:

Category,UserStory,Type,ModuleID,Title,VerifiedInVersion,QuickstartID,Comments,Jira
installing-on-apache-karaf,"As an Evaluator, I want to install Fuse on Karaf, so that I can try it out quickly and discover whether it meets my needs.",assembly,install-karaf-for-evaluator,,,,Evaluator only has access to the kits published on the developer site. Evaluators like to use an IDE and probably have a Windows machine.,
installing-on-apache-karaf,"As a Developer, I want to install Fuse on Karaf, so that I can develop Karaf applications on my local machine.",assembly,install-karaf-for-developer,,,,Developer is probably not that worried about which patch they install. Probably wants to configure Maven properly.,

When you use a spreadsheet to plan modules, there must be columns for Category, Type, Level and ModuleID. However, if you are using the spreadsheet to define the high-level structure of a guide, you will almost certainly want to include the UserStory column as well. Some of the additional columns are preserved as metadata (written into comments in the generated module and assembly files), whilst other additional columns are ignored.

Given a CSV file, sample.csv, you can generate the corresponding modules and assemblies by entering the following command in your content repository:

nebel create-from sample.csv

When using a Google sheet to plan assemblies and modules, the Levels column enables you to specify a nesting level for each module/assembly by using a positive integer, 1…​n. This makes it possible to map out the structure of your book exactly, using arbitrary levels of nesting.

When generating content from a sheet (actually, from an exported CSV file), Nebel automatically generates an accompanying generated_master.adoc file. This file contains the include directives for the top-level items specified in the sheet. This helps you quickly create a skeleton outline of the new book.

Creating modules from an assembly file

The nebel utility can also create modules by scanning an assembly file for AsciiDoc include:: directives and — based on the information available in the include directives — creating corresponding modules that contain template content.

To create modules from an assembly file:

1. Edit an existing assembly file to add some include:: directives for some modules that do not exist yet. For example, in the assemblies/installing-on-apache-karaf/assembly-install-karaf-for-administrators.adoc, you could add the following include directives:

   include::../../modules/installing-on-apache-karaf/proc-downloading-the-latest-karaf-patch.adoc[leveloffset=+1]
   
   include::../../modules/installing-on-apache-karaf/proc-unzipping-karaf-packages.adoc[leveloffset=+1]
   
   include::../../modules/installing-on-apache-karaf/proc-creating-karaf-users.adoc[leveloffset=+1]

2. In the directory that contains the nebel.cfg file, run the following command:

   nebel create-from assemblies/installing-on-apache-karaf/assembly-install-karaf-for-administrators.adoc

   After running this command, you should find three new procedure modules in the modules/installing-on-apache-karaf directory.

Adding symbolic links with Nebel

All content is in the assemblies directory and the modules directory. For publishing a book, the master.adoc file for the book is in another directory, which is a peer to the assemblies directory and modules directory. To generate the book, you need a symbolic link from the book directory to each category directory that contains assemblies or modules that contain the content for the book.

Setting up a book directory for symbolic links

In a book directory, before you add symbolic links to category directories, add an assemblies directory, an images directory, and a modules directory. For example, suppose the name of the book directory is installing-on-jboss-eap. You want the installing-on-jboss-eap directory to contain:

assemblies
images
modules
master-docinfo.xml
master.adoc

Running Nebel to add symbolic links

To run nebel to create symbolic links, the command line has the following form:

nebel book BOOK_DIRECTORY_NAME -c "CATEGORY1,CATEGORY2,...CATEGORYN"

Replace BOOK_DIRECTORY_NAME with the name of the directory that contains the book for which you are adding symbolic links to category directories. In the quotation marks, insert the name of each category directory for which you want symbolic links. For example, the following command adds symbolic links to the directory that contains the book, Installing on JBoss EAP:

nebel book installing-on-jboss-eap -c "installing-on-jboss-eap,maven"

In the installing-on-jboss-eap/assemblies directory, the example command adds symbolic links to:

assemblies/installing-on-jboss-eap
assemblies/maven

In the installing-on-jboss-eap/modules directory, the example command adds symbolic links to:

modules/installing-on-jboss-eap
modules/maven

In the installing-on-jboss-eap/images directory, the example command adds symbolic links to:

images/installing-on-jboss-eap
images/maven

At a later time, if you add a new category in the main assemblies directory or in the main modules directory, you can run the command again and specify only the new category or categories.

Splitting content in one file into assemblies and modules

You can split an existing AsciiDoc file into assemblies and modules using Nebel’s annotation approach. Content that you want to convert must be organized in sections that lend themselves to being assemblies, concept modules, procedure modules, or reference modules.

There are two main use cases for using nebel split:

• You want to convert legacy content to modularized content. In this situation, you iteratively add annotations and run nebel split until you get the output that you want. Thereafter, you maintain the assemblies and modules created by running nebel split. That is, you update the assembly and module files; you no longer update the legacy content.

• Fewer, larger files are preferred over many, smaller, assembly and module files. Usually, this goes with a preference for shorter headings in place of the longer headings that have the distinguishing qualifications required in modular documentation. Upstream communities often prefer larger files and shorter headings. In this situation, each time you fetch the larger files, for example, to reuse the content downstream, there is a script that runs nebel split to create the assemblies and modules. You maintain the larger files; you do not edit the generated assemblies and modules.

The following topics provide information and instructions for doing this:

• Annotating a file for conversion to assemblies and modules

• Generating assemblies and modules from annotated files

• Nebel split annotations reference

• Nebel split command reference

Annotating a file for conversion to assemblies and modules

To prepare a non-modularized AsciiDoc file for conversion to assemblies and modules, add required annotations to the file.

Prerequisites

• You edited the content to be split so that it is easy to distinguish whether a section is an assembly, a concept module, a procedure module or a reference module.

Procedure

1. In a text editor, open the AsciiDoc file to be split.

2. Find the first, top-level heading in the file. For example, the top-level heading might look like this:

   [[debezium-mysql-connector]]
   = Debezium Connector for MySQL

3. Decide which category this content should be in.

4. Determine the content type, that is, whether this section should be an assembly, a concept module, a procedure module, or a reference module. As this is a top-level heading, it is likely that you would want to convert the initial content into an assembly.

5. Insert comments that specify the category and the type.

   For example, suppose that the section should be an assembly in the debezium-using category. Add the following comments (annotations) to the file immediately before the heading:

   // Category: debezium-using
   // Type: assembly
   [[mysql-connector]]
   = Debezium Connector for MySQL

   The default behavior is that the category you specify for the top-level heading applies to all subheadings.

   If you do not specify the category, it defaults to the name of the directory containing the file being split or to a directory named default.

6. Move down to the next section heading, which is most likely a subheading of the preceding heading. In other words, the next section heading is probably indicated by two equals signs, ==.

7. To convert this subheading and its content to a module, decide the content’s type.

8. Add a comment that specifies the content’s type as concept, procedure, or reference. For example:

   // Type: concept
   [[overview]]
   == Overview of MySQL Connector

   It is not necessary to specify a Category annotation for subheadings.

   Any subheadings in this module section do not require annotations. When you run Nebel to split the content, the utility maps any subheadings here to subheadings in the enclosing module. This is the default behavior.

9. Optional. To override the section’s anchor ID, which appears in double square brackets, [[ ]], insert a ModuleID annotation before the heading. For example:

   // Type: concept
   // ModuleID: overview-of-debezium-mysql-connector
   [[overview]]
   == Overview of MySQL Connector

   The generated module or assembly will have the anchor ID that you specify in the ModuleID annotation.

10. Optional. To override the section’s title, insert a Title annotation before the heading. For example:

    // Type: concept
    // ModuleID: overview-of-debezium-mysql-connector
    // Title: Overview of Debezium MySQL connector
    [[overview]]
    == Overview of MySQL Connector

    The generated module or assembly will have the title that you specify in the Title annotation.

11. Proceed through the file, adding Category, Type, ModuleID, and Title annotations before subsection headings as needed.

12. Save and close the file.

Examples

• Debezium connector for MySQL

• Debezium connector for PostgreSQL

Next steps

To generate assemblies and modules for a book, repeat this procedure for each of the book’s files.

Generating assemblies and modules from annotated files

When an AsciiDoc file has modular documentation annotations, you can run Nebel to convert it to assemblies and modules.

Prerequisites

• You added Type annotations, and optionally Category, Title and ModuleID annotations, to one or more AsciiDoc files.

• The Nebel utility is installed.

• You have the latest Nebel updates. If you have not run Nebel in a while, change to your local nebel directory and run git pull.

• The documentation directory in which you want to run nebel meets these conditions:

  • The top-level directory contains a nebel.cfg file. For example, the fuse7/docs directory contains a nebel.cfg file. You can copy nebel.cfg to a directory if you need to.

  • The directory is organized for modularization.

Procedure

1. Open a new shell prompt.

2. Navigate to the directory that contains the nebel.cfg file.

3. To generate assemblies and modules from one AsciiDoc file, run nebel using the following command line format:

   nebel split --legacybasedir LEGACYBASEDIR ANNOTATED_FILE.adoc

   LEGACYBASEDIR specifies the root directory of the file being split, such that the immediate subdirectories of LEGACYBASEDIR implicitly define the default categories.

   ANNOTATED_FILE is the path to the annotated AsciiDoc file.

   For example, if the annotated file is located at ./connectors/mysql.adoc, you could run a command like this:

   nebel split --legacybasedir . connectors/mysql.adoc

   This would store the generated assemblies in the assemblies/connectors directory and the generated modules in the modules/connectors directory. The generated category defaults to connectors, because connectors is the immediate subdirectory of the specified LEGACYBASEDIR (. directory).

   Alternatively, you can specify only the name of the annotated file, for example:

   nebel split upstream/debezium/debezium-1.2/documentation/modules/ROOT/pages/connectors/mysql.adoc

   To generate assemblies and modules from multiple files, use the wildcard, which is a pair of curly braces, {}. For example:

   nebel split --legacybasedir . connectors/{}.adoc

4. Optional. After splitting files, you can run Nebel to fix cross-references that changed because of ModuleID annotations. For example:

   nebel update --fix-links -a upstream/debezium/attributes.adoc,attributes.adoc,attributes-links.adoc -c debezium-using

   -a

   Specifies a comma-separated list of attribute files that that Nebel needs to update references. You must specify the path for the repository’s attributes.adoc and attributes-links.adoc files. If the directory uses any other attributes files, you must specify them as well. This sample command line specifies the attributes.adoc file in the upstream/debezium directory.

   -c

   Specifies the scope of the content in which Nebel updates links. Specify one or more, comma-separated category names. In this example, Nebel fixes links that are in the debezium-using category.

Nebel split annotations reference

In a non-modular AsciiDoc file, a Nebel annotation is a one-line comment immediately before a section heading. There cannot be blank lines between the annotation and the heading it applies to. An annotation comment has the following format:

// ANNOTATION_NAME: ANNOTATION_VALUE

Each annotation that Nebel can interpret and process when splitting large, current files or legacy files into assemblies and modules is described here.

• Category

• Type

• TopicType

• ModuleID

• Title

Category

This annotation specifies the category for the content in the following section. The category is the subdirectory of the assemblies directory or the modules directory in which Nebel will store the generated file. After you specify a category for a particular section, it applies to all of its subsections, and, hence, to all of the assemblies and modules generated from those subsections.

It is typically necessary to set the Category annotation on only the top-level section heading in a file. The rest of the subsections in the file are then implicitly mapped to the same category. If you do not specify the Category annotation, it defaults to one of the following:

• The directory that contains the AsciiDoc file being split, as indicated by the --legacybasedir option in the nebel split command

• The assemblies/default and modules/default directories, which nebel split creates if --legacybasedir was not specified in the nebel split command

For example, given the following Category annotation:

// Category: integration-nebel
// Type: assembly
[id="splitting-content"]
== Splitting content

Nebel creates the assembly-splitting-content.adoc file in the assemblies/integration-nebel/ directory.

Type

This annotation Specifies the kind of module to map the following section to.

Table 1. Descriptions of Type annotation values

Value	Description
assembly	Maps the section to an assembly.
concept	Maps the section to a concept module.
procedure	Maps the section to a procedure module.
reference	Maps the section to a reference module.
continue	Absorbs the section into the preceding module or assembly. It becomes a subsection of the preceding module or assembly.
skip	Skips the section. It does not appear in the generated assembly or module.

If you do not specify a Type annotation for a section, the section becomes a subsection of the nearest enclosing module or assembly.

TopicType

This annotation is an alias of Type. In other words, you can use either the TopicType annotation alias or the Type annotation to specify the module or assembly type.

ModuleID

This annotation specifies a new value for the section’s anchor ID. In the generated module, the ModuleID value that you specify in this annotation replaces the ID in the unsplit source file.

If you want the generated section to have a different ID from the content being split, use this annotation instead of changing the existing ID directly. This makes it possible for Nebel to update cross-references to this section so that they specify the new anchor ID. For example:

// Category: integration-nebel
// Type: assembly
// ModuleID: splitting-content-by-inserting-annotations
[id="splitting-content"]
== Splitting content

With these annotations, Nebel creates the assembly-splitting-content-by-inserting-annotations.adoc file and the anchor ID in the file is

[id="splitting-content-by-inserting-annotations"]

Title

This annotation specifies a new title (heading) for the section. In the generated module, Nebel replaces the heading that is in the file being split with the Title value that you specify in this annotation.

This is useful for keeping short headings upstream and also having longer, descriptive headings downstream. For example:

// Type: concept
// ModuleID: overview-of-splitting-asciidoc-by-using-annotations
// Title: Overview of splitting AsciiDoc by using annotations
[[Overview]]
== Overview

With these annotations, Nebel creates the con-overview-of-splitting-asciidoc-by-using-annotations.adoc file and the heading in the file is

== Overview of splitting AsciiDoc by using annotations

Nebel split command reference

The nebel split command, has the following format:

nebel split [-h] [options] FROM_FILE

Run nebel split -h to display help for the options. Replace FROM_FILE with the name of the .adoc file to split.

Table 2. Descriptions of nebel split command options

Option	Description
-a	One or more attribute files (comma separated) that Nebel needs to resolve links to included files in the assemblies that Nebel generates. Attributes are often in the path names specified in include statements. When Nebel splits content, it must be able to resolve (find) included files.
--conditions	One or more condition names (comma separated) specified in ifdef or ifndef statements in content to be split. Nebel keeps or deletes content tagged with the conditions you specify depending on whether the source specifies ifdef or ifndef. Nebel does this before it splits the content. A common scenario is that you want to delete upstream-only content when you are splitting content into files that will be used downstream. For example, suppose that you tag upstream-only content with ifdef::community[] and you tag downstream-only content with ifdef::product[]. When you run nebel split and specify --conditions product, Nebel keeps content tagged with product and deletes content tagged with community. Now suppose that you specify ifndef::community[] statements to tag content that is downstream-only. To keep content tagged by ifndef::community[] in the generated assemblies and modules, run nebel split with --conditions community. If the content being split contains ifdef or ifndef statements and you run nebel split without specifying the --conditions option, Nebel keeps and splits all content. This is likely to lead to erroneous output.
--category-prefix	Adds the specified prefix to any generated categories, which can be helpful to distinguish categories from one another. For example, suppose that an annotation instructs Nebel to generate some modules in the category getting-started. By running nebel split and specifying --category-prefix debezium, Nebel would add modules to the debezium-getting-started category. In a split annotation, if you specify, for example, // Category: debezium-getting-started and you run nebel split with the --category debezium option, Nebel adds files to the debezium-getting-started category and not to the debezium-debezium-getting-started category.
--legacybasedir	Specifies the root directory of the file(s) being split. Nebel might use this directory to determine the category for the generated files.
--timestamp	Inserts a timestamp in each generated assembly and module file.

Here is an example of a nebel split command:

nebel split -a attributes.adoc --conditions product --category-prefix debezium --legacybasedir upstream/debezium/debezium-$branch/documentation/modules/ROOT/pages upstream/debezium/debezium-$branch/documentation/modules/ROOT/pages/connectors/postgresql.adoc

This command:

• Is executed in the integration/docs directory.

• Specifies the attributes.adoc file that is in the docs directory.

• Instructs nebel split to delete content that is tagged with a condition other than product. For example, content enclosed in ifdef::community[] and endif::community[] statements is deleted.

• Specifies that the names of categories that will contain the generated files begin with debezium.

• Provides the path of the root directory for the file to be split.

• Specifies the path of the file to be split.

Identifying orphan files

You can run Nebel to identify files that are not included in a master.adoc file or an assembly. It is then up to you to determine whether to delete an orphan file or add an include statement for an orphan file.

This is helpful when you regularly run nebel split upon fetching updated upstream content. If you change a Nebel annotation in an upstream file, the next time you fetch the file, nebel split might create a file with a different name but with the same content as an existing file. For example, suppose an upstream file has this annotation:

// Type: concept
// ModuleID: descriptions-of-events

You fetch the upstream file and nebel split generates the con-descriptions-of-events.adoc file, which is included in an assembly. Later, you change the annotation:

// ModuleID: descriptions-of-debezium-events

The next time that you fetch the upstream file, nebel split generates the con-descriptions-of-debezium-events.adoc file and includes this file in an assembly. The con-descriptions-of-events.adoc file is no longer included in an assembly; it is an orphan file that you can delete.

The format for running nebel orphan is:

nebel orphan [-h] [-c CATEGORY_LIST] [-a ATTRIBUTE_FILES]

-h

Displays a help message.

-c CATEGORY_LIST

Replace CATEGORY_LIST with a comma-separated list of categories. Nebel checks for orphan files by evaluating include statements in all master.adoc files and in assemblies and modules subdirectories for only the categories that you specify. If you do not specify the -c option, Nebel checks all categories for orphans.

-a ATTRIBUTE_FILES

Replace ATTRIBUTE_FILES with a comma-separated list of attribute files that Nebel needs to resolve paths in include statements in the categories that the command is checking.

For example:

nebel orphan -c debezium-using -a attributes.adoc,upstream/debezium/attributes.adoc

This command resolves include statements in assemblies that are in the debezium-using category. To do this, Nebel needs the toplevel attributes.adoc file, and it also needed the upstream/debezium/attributes.adoc file.

Renaming or moving files

The nebel mv command lets you move or rename a file (or files) without breaking any include directives. In particular, this subcommand was originally implemented to assist with renaming modular file prefixes. For example, consider a collection of procedure modules whose file names start with p_. To change that prefix to proc- you can rename the files by running a command like this:

nebel mv modules/getting-started/p_{}.adoc modules/getting-started/proc-{}.adoc

The nebel utility updates include directives as well as links that contain the file names that are being changed.

Backwards-incompatible change to modular file prefixes

Prior to Nebel version 2, Nebel assumed that the underscore, , was the separator for modular file prefixes. For example, file names had prefixes such proc, con_, and ref_. It was possible to customize the prefixes, by setting some properties in the nebel.cfg file, but it was not possible to change the separator to be anything other than an underscore.

Starting with Nebel version 2, however, it is possible to customize file prefixes, including the separator character, by editing settings in the nebel.cfg file. For example:

[Nebel]
dir.assemblies = assemblies
dir.modules = modules
prefix.assembly = assembly-
prefix.procedure = proc-
prefix.concept = con-
prefix.reference = ref-

Nebel versioning

Nebel now supports a version flag, which you can use to check the particular version you are using, for example:

nebel -v
Nebel 2.1.x (dev release)

Here is a recent version history:

• 1.0.0 — First numbered version (from April 4, 2020), uses the old convention for modular file prefixes (underscore separator is hardcoded).

• 2.0.x — Backwards-incompatible update, uses the new convention for modular file prefixes (separator character is part of the customizable prefix, thus enabling you to use a hyphen separator).

• 2.1.x — Supports the new nebel split subcommand.

Nebel Python interpreter

The nebel utility is coded for the Python 2 interpreter and does not work with Python 3. On May 15, I modified the nebel binary, so that it calls the Python 2 interpreter explicitly (instead of calling the ambiguous Python executable). This ensures that nebel also runs correctly on recent Fedora and RHEL releases. In the long run, nebel will need to be updated for Python 3.