Building Blocks and Visual Representation

[JochenKienzle]

Context and Problem Statement

Like in arc42, building blocks are used to define and decomposition of a system based on requirements, constraints, quality goals, etc. I want to use sphinx-needs to define these building blocks to link these artifacts together. Beside the 'textual' definition of a building block there is also the need of views like an overview but also specific deployments.

I want to use:

• sphinx-needs to define different level of building blocks via types
  • abb: architectural building block
  • sbb: solution building block
• c4-plantuml to create a visual representation of these building blocks
  • c4 container => abb
  • c4 component => sbb

Example for building block ins sphinx-needs

This is a simplified example of building blocks in sphinx-needs:

.. abb:: Logging
   :id: ABB_Logging
   :provides: <list of requirement ids>
   :considers: <list of constraint ids>
   :provided logical interfaces: <list of provided logical interfaces>
   :required logical interfaces: <list of logical interfaces>

   <textual description>

.. sbb:: Logger X
   :id: SBB_LoggerX
   :implements: ABB_Logging
   :provided interfaces: <list of provided interfaces>
   :required interfaces: <list of interfaces>

   <textual description>

Example for visual representation

• puml-file which defines c4-containers which represents e.g. the ABB_Logging and shows their provided and required interfaces
• puml-file which defines c4-component which represents e.g. the SBB_LoggerX and shows their provided and required interfaces
• puml-file which provides the overview of all architectural building blocks (interfaces are connected)
• puml-file which provides a specific deployment using solution building blocks (interfaces are connected)

Ideas for linking / integrating sphinx-needs and visual (plantuml) building blocks

Proposal 1: Generate plantuml 'model' out of sphinx-needs

Concept:

• sphinx-needs definition (using abb & sbb types) defines the building block and links all artifacts
• sphinx-needs definition is used to generate 'base' plantuml files which visualize these building blocks
• further plantuml files for specific deployments and other views using includes from the 'base' plantuml files

Pro/Cons:

• naming of c4 model artifacts are generated and are aligned with definitions in sphinx-needs definition
• generator from sphinx-need types to puml is needed
• centralized puml files are needed

Proposal 2: Describe plantuml content in sphinx-needs (via generator)

Concept:

• sphinx-need definition (using abb & sbb types) defines the building blocks and links all artifacts
• plantuml content is parsed and add links to sphinx-needs types like ABB und SBB incl. checks of e.g. interfaces
• each plantuml defines a sphinx-needs object like "VIEW_A" which contains the links

Pros/Cons:

• Additional overhead via VISUAL sphinx-need types
• Each plantuml file defines its own content and sphinx-needs tries to resolve the links
• Views are integrate into sphinx-needs
• Meta data of the sphinx-needs types could be used to ensure consistency of names, interfaces, etc. but additional checks are needed
• No central plantuml 'model' file is needed

Open points / Questions

1. How to generate links/objects from plantuml to sphinx-needs or vice versa?
2. What are the common concept to include artifacts which are defined outside of sphinx-needs?

I got some hints to use open-needs which might be an answer for these questions.

I would happy about any feedback from you :-)

[danwos]

Thanks for the great ideas.

So as far as I got you correct, we have 3 main problems / challenges:

1. A C4 specific representation for needflow is needed.
2. Different views of the same data / models
3. Get a PlantUML representation, which is a mix of data from Sphinx-Needs and other sources (maybe manual defined).

Solution proposals:

1. A C4 specific representation for plantum is needed
This may be already possible (but maybe ugly).

1. You could use needs_flow_config to define the needed configuration to get C4 into Plantuml.
2. The config of needs_types has a style parameter, which takes a PlantUML model name. It's just a string and gets copied 1:1 into the plantuml-model-code. So if PlantUML can handle the name, everything is fine.

2. Different views of the same data / models
This may also be possible by:
a) Defining filters, which contains all needed objects, which shall be shown
b) Defining the link-types, which shall be shown.

But I see here room for improvement, e.g. if a "connection/edge" is represented by a Sphinx-Needs object and not only by a link-type.
Something like this is e.g. needed for needsequence, where a Sphinx-Needs object defines the "message" between two participants. But in the graphical representation this "message" is more or less just a line/connection.

3. Get a PlantUML representation, which is a mix of data from Sphinx-Needs
Maybe this can be realized by allowing needflow to have a content, which must be PlantUML and is added before/after the generated code of Sphinx-Needs.
So this manual code would have access to the PlantUML objects defined by Sphinx-Needs. It could introduce new objects and even link a newly created object with a Sphinx-Needs object.

This could also be combined with 2. Different views of the same data / models.
But I'm not sure, if this already helps for all use cases.

I maybe have also an answer to one of your open questions:
How to generate links/objects from plantuml to sphinx-needs or vice versa?
If you mean with links real, clickable HTML-Links, then you could set the output format of PlantUML to svg or svg_img (Can't just remember which was the working one.).
In the generated svg-image all PlantUML blocks defined by Sphinx-Needs will be a link to the location of the original Sphinx-Needs object in the docs.
But this output format has one drawback, it can not be scaled, otherwise it loses this functionality (This is based on some browser-specs and can't be changed. Last checked mid 2021.)

I'm pretty sure I missed to comment on some of your ideas.

[JochenKienzle]

Hi @danwos,
many thanks for your quick reply.

Regarding 1:
I was not aware about the needs_flow directive. If I understand it correctly, it allows to define plantum syntax directly in inside the sphinx-needs description but does it provide automatically generation of objects and links based on the sphinx-needs objects/data (e.g. automatically generate plantuml out of sphinx-needs types like requirements and use options to define the links relationships between them)?

Regarding 2 and 3:
Based on your feedback, I think we can summarize it into two to use cases:

1. (Automatically) transformation of sphinx-needs objects to plantuml
2. A way to generate needed sphinx-needs data out of plantuml

The first use case is to define the building blocks and create a base in the 'other model'. Maybe needflow could be used / extended here.

The second use case would allow to use 'the other' representation to work further with it (e.g. views of specific deployments,...) which can then link back to sphinx-needs. For that a parser of the plantuml-model is needed.

[danwos]

Hi @JochenKienzle,

Regarding 1:
no, needsflow generates PlantUML code out of a selection of Sphinx-Needs objects. And it uses the link-options to define/show the links between them. You can't define the PlantUML code by hand. It's all automated.
However, the code itself is not available, as it gets transformed to the final image during a Sphinx build.
(But you can see it by setting :debug: flag -> docs)

But I have the feeling that we could model all your architecture-needs with Sphinx-Needs objects and use needflow to get the needed graphical views out of it. Maybe for 1-2 use cases, some additional feature implementations are needed, but I think that this is possible.

Regarding 2 and 3:
For 1) I agree and we need to find a way to make the calculated PlantUML-code extensible by own code. E.g. by the content area of the needflow directive.

For 2) I'm not sure if I like the idea of having a PlantUML definition as the source of a Sphinx-Needs object.
This has multiple reasons:

1. Defining PlantUML objects can be done in different ways and with different styles, colors, .... . So it would be hard and time-consuming to maintain a parser for this.
2. There is no PlantUML API / object db, which we could use and let PlantUML do the work of parsing ....
3. There can be multiple views of the same data. So which one is the valid one? Does the information of different views need to be combined to define the related Sphinx-Needs object?

I think the way PlantUML -> Sphinx-Needs object has too many inaccuracies and too many ways how stuff could be defined.
I would like to stay with Sphinx-Needs object -> PlantUML and provide more features to allow different views/layouts or to combine data from different sources for a common PlantUML model.
But I'm open for any discussion :)

[JochenKienzle]

Many thanks for the clarification of needsflow.

Regarding the other points and proposals I need more time to think and check it against our use cases.

[JochenKienzle]

I had some further thoughts based on your input.

I have following idea in my head:

1. sphinx-needs defines the building blocks for further plantuml modelling (export to plantuml; nothing new so far)
2. plantuml is based on the building blocks imported from sphinx-needs. Sphinx-needs is then only used to provide the final documentation which means it would reference/include these additional plantuml files but it won't "import" them.

Sphinx-needs would have two roles here:

1. Define building blocks and link them to related sphinx-needs artifacts (requirements etc.)
2. Establish traceability to external artifacts and generate 'documentation'
   • By providing an export which act as the base for the subsequent tooling
   • Keep references to artifacts to the other / external tool

What do you think about this approach?

[danwos]

This sounds like a way to go :)

But short questions about this part "Sphinx-needs is then only used to provide the final documentation which means it would reference/include these additional plantuml files".
What are the additional plantuml files? What is their content?

Do you want to create these files by exporting Sphinx-Needs data into a plantuml file, so that it can be extended and then this file gets loaded by e.g. .. uml:: file.puml.
If yes, then my questions is how to keep this file automatically in sync or how to not override manually added content during a new export?

Another approach would be something like this:

.. needflow::
   :filter: id in ["need_1", "need_2"]
   
   ' my additional plantuml code
   card "my_req" as req
   
   req -> need_1

This would then lead to this plantuml:

node "need_1"
node "need_2" 

need_1 -> need_2

' my additional plantuml code
card "my_req" as req
   
req -> need_1
 

So no export of Sphinx-Needs data to a file.
Instead the PlantUML Code is a mix of generated code based on a given filter and a manual written part.

[JochenKienzle]

Hi @danwos

the roles of both tools could be:

• sphinx-needs: handling of requirements and do architectural break down to building blocks; provide html documentation
• plantuml: graphical representation and modeling of deployments etc. (without using sphinx-needs types but based on the provided building blocks)

The sync point between both tools would be an exported plantuml file which contains the building blocks defined via sphinx-needs. This file should be generated out of sphinx-needs. Plantuml is then further used to visualize and to define models based on plantuml (e.g. deployments, releases,...). To avoid duplications and re-definitions of these building blocks, these building blocks should be always included in other plantuml files to stay in sync.

The plantuml tooling part could be put into another doc-toolchain to generate its documentation and it would not create any new content in sphinx-needs (clear separation) but sphinx-needs could be reused here to generate the documentation for the plantuml part as well.

[danwos]

Ahh, I see and I need to get more into this to understand fully the requirements. :)
I understand the way the final PlantUML code gets constructed.
But I'm not sure if the final "merge" should be done in an extra PlantUML file, which model-basement got once exported.

However, in your case, a small script would be enough, which uses an existing needs.json file and the filtering rule to create a single PlantUML file, which can be reused/extended.
So I see (currently) no need to extend Sphinx-Needs itself and this little script should be a new small project, called "Needs2PlantUML".

[twodrops]

@JochenKienzle and @danwos Thanks for the great discussion here,

My short comments here.

• I am not sure if we should extend sphinx-needs as a semantic modeling language. It shines well where we have 80% text (a markup) and 20% semantic (using needtypes/directives in rST) and not the other way around.
• The modeling capabilities of sphinx-needs in the direction of semantic modeling is currently limited because of the current design of defining types. That, is need_types and global character of need_extra_options. That is the need_extra_options are not related to the need_types directly. These are very important when it comes to semantic modeling, else semantic meta-model consistency can be ensured only by validations. These could be overcome with the upcoming open-needs.
• For supporting "building blocks", which would only a part of architecture modeling, the needflow could be reused. A project I know does this already. If a different visualization with C4 is needed, a needflowc4 could be created.
• The power of semantic modeling, IMHO, should be left with the tools which are designed for it. That is within, PlantUML/C4 (with the limited semantic modeling capabilities), SysML2, custom textual DSLs etc. and the information imported into Sphinx-Needs for traceability purposes only. The architectural validation, IMHO, should run outside the scope of sphinx-needs. In the end, docs-as-code is for documentation generation and not for code-generation, architecture validation etc.

[danwos]

Maybe the already existing concept of #443 can help us here as well, as it allows, even more, PlantUML manipulations/extensions with data from Sphinx-Needs.