Make null vs Freed Object behavior more consistent

[dalexeev]

• See also Overhaul equality-checking and type-coercion #10553.
• See also Make objects non-nullable by default #11054.

Origins

"The most important data type in Godot" is Variant. All values ​​in GDScript are Variant. It has two fields: type (TYPE_NIL, TYPE_OBJECT, etc.) and content (a union). Also Godot/GDScript does not have a garbage collector. This means that objects can be destroyed before all references to the object become unreachable. Therefore, we have several possibilities to have an "invalid object" (null-like values):

• null (type is TYPE_NIL, content is zero)
  • This is a special value intended to represent empty data.
• Object#null (type is TYPE_OBJECT, content is zero)
  • This value is essentially obtainable by mistake. In some cases, engine methods return Object#null (Ref<T>()) instead of null Variant(). For example, Array.get_typed_script().
• Freed Object (type is TYPE_OBJECT, content is non-zero, after the object is destroyed)
  • This is indistinguishable from valid objects without checking the ObjectID in OBjectDB. The Variant data does not change when the pointed object is removed from memory.

Problems

While working with the engine, you come across various examples of value comparisons and type conversions (both explicit and implicit):

• Stringification
  • There are two main ways to represent a value as a string: str() and var_to_str(). Usually var_to_str() gives you more details, but in the case of null-like values ​​this is not true, they are all converted to null because var_to_str() is designed for text serialization.
  • See also Previously freed objects shown as <null> in the debugger godot#92632.
• Booleanization
  • There is an explicit bool() constructor, but in 4.x it supports few types.
    • Related: GDScript No constructor of "int" matches the signature "int(String)" godot#44407 (comment).
  • There are special methods such as is_empty(), is_valid(), is_null(). That last one is confusing.
    • See Callable: Not nullable, inconsistent behaviour godot#72209.
    • See Callable as function argument is not nullable godot#72350.
  • In the context of conditions (if, elif), loops (while) and logical operators (and, or, not), values ​​are cast to bool implicitly.
• Comparison
  • The == operator allows you to compare values ​​in a "reasonable" way. In some cases, similar values ​​of different types (such as 1 and 1.0) are considered equal, in other cases this results in an error.
  • See also Allow for comparing values of different types #4571.
  • See also Add strict and safe equality operator === for comparing references and variants (implemented as is_same() method) #6092.
• Strict Comparison
  • In 4.x we have a global function is_same() for strict comparisons.
• Matching
  • The match statement has special behavior that differs from both operator == and is_same().
• Content Testing
  • GDScript has the in keyword.
  • There are also methods Array.has(), Dictionary.has(), Dictionary.find_key() and others.
• Type Checking
  • There are several ways to check a type: the is keyword, global functions typeof() and is_instance_of().
• Type Casting
  • There is explicit type casting using the as keyword, constructors, special methods, global functions convert() and type_convert(). There is also implicit type casting in a typed context (assignment, argument passing, etc.).
• Some of the operations above are considered errors for Freed Object, but not for null.
  • See GDScript: Assume freed object to be of any Object type godot#87296.

Value comparisons and type conversions in 3.x and 4.x

3.5.3

Stringification
---------------
str(value)                    Null   [Object:null]  [Deleted Object]
var2str(value)                null   null           null

Booleanization
--------------
true if value else false      False  False          True
true if not value else false  True   True           False
is_instance_valid(value)      False  False          False

Comparison
----------
value == null                 True   True           False
value == other_obj_null       True   True           False
value == other_freed          False  False          False

Type Checking
-------------
typeof(value) == TYPE_NIL     True   False          False
typeof(value) == TYPE_OBJECT  False  True           True
value is Object               False  True           True

4.2.2 -> 4.3 dev 6

Stringification
---------------
str(value)                      <null>         <Object#null>  <Freed Object>
var_to_str(value)               null           null           null

Booleanization
--------------
true if value else false        false          false          true -> false
true if not value else false    true           true           true
is_instance_valid(value)        false          false          false

Comparison
----------
value == null                   true           true           true -> false
value == other_obj_null         true           true           true -> false
value == other_freed            true -> false  true -> false  true -> false

Strict Comparison
-----------------
is_same(value, null)            true           false          false
is_same(value, other_obj_null)  false          true           false
is_same(value, other_freed)     false          false          false

Type Checking
-------------
typeof(value) == TYPE_NIL       true           false          false
typeof(value) == TYPE_OBJECT    false          true           true
value is Object                 false          false          Error

Summary

3.x -> 4.0

                                null           Object#null    Freed Object

true if value else false        false          false          true
true if not value else false    true           true           false -> true
is_instance_valid(value)        false          false          false

value == null                   true           true           false -> true
value == other_obj_null         true           true           false -> true
value == other_freed            false -> true  false -> true  false -> true

typeof(value) == TYPE_NIL       true           false          false
typeof(value) == TYPE_OBJECT    false          true           true
value is Object                 false          true -> false  true -> Error

For some reason, maybe by mistake, the following changes occurred during the 4.x implementation:

1. if not freed now evaluates to true, just like if freed. Obviously it's a bug, if one is true, then the other should be false.
2. Freed Object now equals null and Object#null.

Proposal Options

Option 1. Restoring 3.x Behavior

This makes sense, but is less beginner friendly and breaks compatibility with 4.0-4.2. In addition, there are other changes compared to 3.x. Finally, 4.x has the global function is_same(). So the of 3.x behavior in itself cannot justify this option.

str(value)                      <null>         <Object#null>  <Freed Object>

true if value else false        false          false          true
true if not value else false    true           true           true -> false
is_instance_valid(value)        false          false          false

value == null                   true           true           true -> false
value == other_obj_null         true           true           true -> false
value == other_freed            true -> false  true -> false  true -> false

Option 2. Intuitive and Beginner Friendly

This option breaks compatibility to a lesser extent, it only fixes already broken behavior. This also makes the use of the more verbose is_instance_valid() optional.

str(value)                      <null>         <Object#null>  <Freed Object>

true if value else false        false          false          true -> false
true if not value else false    true           true           true
is_instance_valid(value)        false          false          false

value == null                   true           true           true
value == other_obj_null         true           true           true
value == other_freed            true           true           true

Option 3. godotengine/godot#73896 (reverted)

Note that godotengine/godot#73896 mixes the two above options, which seems like a less consistent move.

str(value)                      <null>         <Object#null>  <Freed Object>

true if value else false        false          false          true -> false
true if not value else false    true           true           true
is_instance_valid(value)        false          false          false

value == null                   true           true           true -> false
value == other_obj_null         true           true           true -> false
value == other_freed            true -> false  true -> false  true -> false

Comparison Styles

I compared existing options for comparing null-like values ​​used in various places in the engine. We should take this into account when making a decision.

1. Intuitive

Treat null, Object#null and Freed Object as equal. This is the most intuitively expected behavior for users coming from languages ​​with garbage collection.

• Operator == for null-like values in 4.0-4.2, 4.3 dev 1-5, and current master

             | null | Object#null | Freed Object
-------------|------|-------------|--------------
null         | true | true        | true
Object#null  | true | true        | true
Freed Object | true | true        | true (always)

2. Strict

Treat values ​​as different if they differ by Variant.Type and/or content (for reference types, the content is the reference).

Note that for dictionary keys this is the only option, since the keys must be unique. If you add two object keys to a dictionary and then free the objects, you should not end up with two equal (in Dictionary terms) keys in one dictionary.¹

• is_same()
• Dictionary.has()
• Operator == for dictionaries (keys)
• match for dictionaries (keys)

             | null  | Object#null | Freed Object
-------------|-------|-------------|-------------
null         | true  | false       | false
Object#null  | false | true        | false
Freed Object | false | false       | true (same)
             |       |             | false (diff)

3. By Variant.Type

Treat null and Object#null/Freed Object as different (because they have different types: TYPE_NIL and TYPE_OBJECT), but Object#null and Freed Object as equal (because both are TYPE_OBJECT and point to an invalid instance).

• Array.has()
• Operator == for arrays
• Dictionary.find_key() (find key by value)
• Operator == for dictionaries (values)
• match for null-like values
• match for arrays
• match for dictionaries (values)

             | null  | Object#null | Freed Object
-------------|-------|-------------|--------------
null         | true  | false       | false
Object#null  | false | true        | true
Freed Object | false | true        | true (always)

4. By ObjectID

Treat null and Object#null as equal (because they have a zero ObjectID), but null/Object#null and Freed Object as different (because Freed Object has a non-zero ObjectID). Two Freed Objects are equal if they have equal ObjectIDs.

• Operator == for null-like values in 3.x, 4.3 dev 6 - beta 2
• Make freed object different than null in comparison operators (reverted) godot#73896

             | null  | Object#null | Freed Object
-------------|-------|-------------|-------------
null         | true  | true        | false
Object#null  | true  | true        | false
Freed Object | false | false       | true (same)
             |       |             | false (diff)

Footnotes

1. Note that the Dictionary treats String and StringName as the same. But it's safe because, unlike objects, two "equal" String/StringName can never become different. ↩