[WIP] Choice rule payload validation

[kbrock]

alt to #187

Depend upon:

• Pass full path name to State.new for better errors #229
• [WIP] Catch and display path errors #246
• Choice rule payload validation path #253
• Split out ChoiceRule spec to be similar to other specs #254
• In ChoiceRule::Data, parse compare key and variable  #257
• Use Numeric over Integer || Float #264
• Implement "IsNumeric": false #266
• Support choices that do not have a Default defined #267
• Choice: rename value to compare_predicate #273
• More granular compare_key and determine path at initialization time #274
• Choice State enforces type checking #276
• Use send for choice operations #295

Overview

Choice does not quite behave the same as the AWS States Language reference implementation.

If you specify a Path that points to nothing or the wrong type, aws raises a States.Runtime. In truth, the type checking is not the most consistent, but the missing path is always raised.

Changes:

• Validate Next points to a valid state.
• Validate Variable and compare key values are the correct data type.
• Raise exceptions when Variable or compare key path values not not found or the wrong data type. It used to throw ruby exceptions.
• Fix "IsPresent": true to detect values present rather than not null.
• All "Is*" comparisons now support true and false values.
• Support Choice with no Default provided.

[kbrock]

update:

• cop: returning false instead of nil for bad lhv
• cop: space in spec
• rebased

[kbrock]

wip: pre-compiling operations

[kbrock]

update:

• compiled expression into command tree
• changed key comparison to not look for String, but rather (String)(GreaterThan)(Path). It has a few false positives, (IntegerMatchesPath) but they will all probably work.

This is a lot more involved but a lot more strict/stringent

update:

• cops: hash rocket, indent, annotation
• no longer escaping matches up front (can roll back if others want)

[kbrock]

rubocop: command line and web have different opinions on whether a regex is freezable. one complains no matter what code I use here.

un-wip: this seems to work

[kbrock]

update:

• rebase
• simplified down choice class lookup code

update:

• added freeze back in

[Fryguy]

Can you add specs around these new classes? wondering if would catch the invalid namespace thing I found.

[kbrock]

update:

• change class reference

I commented out every line there, and the specs fail. So every one of those classes are tested

[kbrock]

Discussion:
Instead of using a command pattern, we can change this to store the operation so we wouldn't need all these classes.

e.g.:

OPERATIONS = { "StringEquals" => "==" }
lhs.send(OPERATIONS[choice_value], rhs)

[Fryguy]

I think this would read better if the match was decomposed or perhaps if named captures were used instead of using values[2], values[3], etc.

Some other quetsions

• where is the prefix used (the (String|Numeric|Boolean|Timestamp) part)? It seems to be ignored here.
• The caller does compare_key, klass = klass_params(payload), but this method returns 3 items, so is that a bug? Where did the 3rd param go with the path, and why isn't it being used?

[kbrock]

the current pattern for the states/command objects to parse the payload themselves.
Here, we need to parse the string to

The old code ignored the first part, so I continued to ignore it. I can see doing type checks or conversions.

returning 2 vs 3 is probably a bug. thanks

[kbrock]

1. yes, we ignore the type prefix part and don't currently do data type conversion. (not sure if that is needed)
2. changed this code. think all set.

[Fryguy]

Discussion: Instead of using a command pattern, we can change this to store the operation so we wouldn't need all these classes.

e.g.:

OPERATIONS = { "StringEquals" => "==" }
lhs.send(OPERATIONS[choice_value], rhs)

Maybe, but this all feels overly complicated to me personally. I don't see why each of these can't be a simple method (or even just the original code the way it was). Just curious what problem you were trying to solve. I could see creating the classes to encapsulate a true? + valid? pair, but all of the new classes just have a single method.

[kbrock]

sorry - this is for previous commit

update:

• rebase
• dropped comparison classes and put into an operation's hash.
• No longer modify parent ChoiceRule class
• storing path or value in ref to simplify the initialize code

[kbrock]

update

• went back to all the is_{type} methods
• dropped all separate classes
• more pedantic at build time (fewer invalid cases go through

[kbrock]

update:

• rebase
• renamed UNARY_RULE to TYPE_CHECK
• renamed BINARY_RULE to OPERATION
• added some comments around @type and @path variables

just superficial changes. Code hasn't changed

[kbrock]

Before:

• fetches a value, and verifies what ever came out is not null

After:

• fetches a value and verifies it existed in the input hash

[kbrock]

Before: "IsString" => "*" verified it is a string.
After: "IsString" => true works that way, but "IsString" => false does the opposite
(across all checks)

[kbrock]

(moved over from ChoiceRule since this is data specific)
Before:

• convert @variable to a Path at runtime and fetch value.
  After:
• @variable is a Path`, so bad Path detected at initialization time
• Path not present in input is a runtime error
• Data type of input is checked.

[kbrock]

Yea, this got more complicated.

After:

• Can detect bad compare values at initialization time.
• Can detect bad paths at initialization time.
• Store type checking information (only for operations. bad type in the type check are informational (true false) rather than errors.

[kbrock]

This one change has big consequences.

Before: A path pointing to nothing would just treat the value as nil
After: A path pointing to nothing is a runtime error

[kbrock]

This was detected because tests running against a state would work, but running against a workflow would not.

Before:

• Errors needed to be caught by each State
• Errors resulted in a state that was not complete.
• Errors did not always get their way into output.

After:

• runtime errors always set the context accordingly.

[kbrock]

The code for finish hasn't changed, it just now resides in mark_finished.

Reasoning: If State throws an error, it almost always threw it in finish. When an error is thrown, super.finish is not called. This means no logging and FinishedTime is never set -- so the state looks like it is not finished.

Now, we catch errors thrown, and ensure super.finish is called. Easiest way to do this is to move that code into a separate method -- mark_finish.

[kbrock]

WIP: piece by piece pulling out PRs from this

[kbrock]

update:

• fix guard clause - ensured bad operators found in initializer not runtime

update:

• fix } spacing cop

[miq-bot]

Checked commit kbrock@a786b40 with ruby 3.1.5, rubocop 1.56.3, haml-lint 0.51.0, and yamllint
1 file checked, 0 offenses detected
Everything looks fine. 🍰