There are two types of transformation rules available:
-
Regex Transform
- Type:
"regex"
- Parameters:
pattern
: The regular expression pattern to matchreplace
: The replacement string
- Description: Applies a regular expression replacement on string values in the specified column.
- Type:
-
Mask Transform
- Type:
"mask"
- Parameters:
mask_char
: The character to use for masking
- Description: Masks the content of string values, keeping the first and last characters visible and replacing the rest with the specified mask character.
- Type:
Filtering rules use various comparison operators to determine whether a row should be included in the output. The available operators are:
- Equality (
"eq"
) - Inequality (
"ne"
) - Greater Than (
"gt"
) - Less Than (
"lt"
) - Greater Than or Equal To (
"gte"
) - Less Than or Equal To (
"lte"
) - Contains (
"contains"
)
Both transformation and filtering rules share these common properties:
type
: Specifies whether it's a "transform" or "filter" rule.column
: The name of the column to apply the rule to.operations
: An array of operations (INSERT, UPDATE, DELETE) to which the rule should be applied. If not specified, it applies to all operations.allow_empty_deletes
: A boolean flag that, when set to true, allows the rule to process delete operations even if the column value is empty.
- The rules support various data types, including integers, floats, strings, timestamps, booleans, and numeric (decimal) values.
- For filtering rules, the comparison is type-aware, ensuring that values are compared appropriately based on their data type.
- The
contains
operator for filtering only works on string values. - Transformation rules currently only work on string values. If a non-string value is encountered, the transformation is skipped and a warning is logged.
To use these rules, you would define them in a YAML configuration file and specify the path to this file using the --rules-config
flag when running pg_flo
. The exact structure of the YAML file should match the rule properties and parameters described above.