Twister2 Configuration Standards

Verbose Properties

Every configuration property should start with the prefix

twister2

Then, it should have the name of the component.

twister2.network

Then it should have the property name. For example here is a property with a name shuffle.memory.limit in the network related configurations.

twister2.network.shuffle.memory.limit

If we specify this property, it is common to both streaming and batch environments. If we want to have a specific property for streaming or batch environment we can add stream or batch add the end. For example, the following property only applies to the stream environment.

**twister2.network.shuffle.memory.limit.stream **

Now if we want this to be applied to a specific operation, we need to add the operation name at the end. The following property only applies to stream keyed-gather operation.

twister2.network.shuffle.memory.limit.stream.keyed-gather

Now in the configuration files, all three properties can be present. In that case, the specific property takes precedence.

twister2.network.shuffle.memory.limit: 1000 twister2.network.shuffle.memory.limit.stream: 10 twister2.network.shuffle.memory.limit.stream.keyed-gather: 100

Also, every property in the files should have a clear block of description.

Hierarchical properties

We can use properties like below for stream and op specific properties as well. Without using verbose properties as described above. The verbose properties above will take precedence always to the same level hierarchical properties.

batch: twister2.network.shuffle.memory.limit: 100 keyed-gather: twister2.network.shuffle.memory.limit: 1000

For example in the above configuration file keyed-gather will get 1000 as the property value. Now if we specify

twister2.network.shuffle.memory.limit.stream.keyed-gather: 10000

in the same configuration file, it will take precedence.