Fix KeyError when deleting a field with unqiue=True #365

gck123 · 2024-11-14T10:39:05Z

Fixes #364

henadzit · 2024-11-18T20:22:17Z

@gck123 can you please add a test exposing the issue?

gck123 · 2024-11-20T02:13:15Z

@gck123 can you please add a test exposing the issue?

Adding these snippets to the specified places and then executing tests.test_migrate.test_migrate will reproduce the KeyError

tests.models add model Tasks

class Tasks(Model):
    id = fields.IntField(pk=True)

tests.old_models add model Tasks

class Tasks(Model):
    id = fields.IntField(pk=True)
    name = fields.CharField(max_length=100, unique=True)

tests.test_migrate replace old_models_describe

old_models_describe = {
    "models.Tasks": {
        "name": "models.Tasks",
        "app": "models",
        "table": "tasks",
        "abstract": False,
        "description": None,
        "docstring": None,
        "unique_together": [],
        "indexes": [],
        "pk_field": {
            "name": "id",
            "field_type": "IntField",
            "db_column": "id",
            "python_type": "int",
            "generated": True,
            "nullable": False,
            "unique": True,
            "indexed": True,
            "default": None,
            "description": None,
            "docstring": None,
            "constraints": {
                "ge": -2147483648,
                "le": 2147483647
            },
            "db_field_types": {
                "": "INT"
            }
        },
        "data_fields": [
            {
                "name": "name",
                "field_type": "CharField",
                "db_column": "name",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": True,
                "indexed": True,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {
                    "max_length": 100
                },
                "db_field_types": {
                    "": "VARCHAR(100)"
                }
            }
        ],
        "fk_fields": [],
        "backward_fk_fields": [],
        "o2o_fields": [],
        "backward_o2o_fields": [],
        "m2m_fields": []
    },
    "models.Category": {
        "name": "models.Category",
        "app": "models",
        "table": "category",
        "abstract": False,
        "description": None,
        "docstring": None,
        "unique_together": [],
        "indexes": [],
        "pk_field": {
            "name": "id",
            "field_type": "IntField",
            "db_column": "id",
            "python_type": "int",
            "generated": True,
            "nullable": False,
            "unique": True,
            "indexed": True,
            "default": None,
            "description": None,
            "docstring": None,
            "constraints": {"ge": 1, "le": 2147483647},
            "db_field_types": {"": "INT"},
        },
        "data_fields": [
            {
                "name": "slug",
                "field_type": "CharField",
                "db_column": "slug",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 200},
                "db_field_types": {"": "VARCHAR(200)"},
            },
            {
                "name": "name",
                "field_type": "CharField",
                "db_column": "name",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 200},
                "db_field_types": {"": "VARCHAR(200)"},
            },
            {
                "name": "created_at",
                "field_type": "DatetimeField",
                "db_column": "created_at",
                "python_type": "datetime.datetime",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"readOnly": True},
                "db_field_types": {
                    "": "TIMESTAMP",
                    "mysql": "DATETIME(6)",
                    "postgres": "TIMESTAMPTZ",
                },
                "auto_now_add": True,
                "auto_now": False,
            },
            {
                "name": "user_id",
                "field_type": "IntField",
                "db_column": "user_id",
                "python_type": "int",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": "User",
                "docstring": None,
                "constraints": {"ge": 1, "le": 2147483647},
                "db_field_types": {"": "INT"},
            },
            {
                "name": "title",
                "field_type": "CharField",
                "db_column": "title",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": True,
                "indexed": True,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 20},
                "db_field_types": {"": "VARCHAR(20)"},
            },
        ],
        "fk_fields": [
            {
                "name": "user",
                "field_type": "ForeignKeyFieldInstance",
                "python_type": "models.User",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": "User",
                "docstring": None,
                "constraints": {},
                "raw_field": "user_id",
                "on_delete": "CASCADE",
            }
        ],
        "backward_fk_fields": [],
        "o2o_fields": [],
        "backward_o2o_fields": [],
        "m2m_fields": [
            {
                "name": "products",
                "field_type": "ManyToManyFieldInstance",
                "python_type": "models.Product",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {},
                "model_name": "models.Product",
                "related_name": "categories",
                "forward_key": "product_id",
                "backward_key": "category_id",
                "through": "product_category",
                "on_delete": "CASCADE",
                "_generated": True,
            }
        ],
    },
    "models.Config": {
        "name": "models.Config",
        "app": "models",
        "table": "configs",
        "abstract": False,
        "description": None,
        "docstring": None,
        "unique_together": [],
        "indexes": [],
        "pk_field": {
            "name": "id",
            "field_type": "IntField",
            "db_column": "id",
            "python_type": "int",
            "generated": True,
            "nullable": False,
            "unique": True,
            "indexed": True,
            "default": None,
            "description": None,
            "docstring": None,
            "constraints": {"ge": 1, "le": 2147483647},
            "db_field_types": {"": "INT"},
        },
        "data_fields": [
            {
                "name": "label",
                "field_type": "CharField",
                "db_column": "label",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 200},
                "db_field_types": {"": "VARCHAR(200)"},
            },
            {
                "name": "key",
                "field_type": "CharField",
                "db_column": "key",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 20},
                "db_field_types": {"": "VARCHAR(20)"},
            },
            {
                "name": "value",
                "field_type": "JSONField",
                "db_column": "value",
                "python_type": "Union[dict, list]",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {},
                "db_field_types": {"": "TEXT", "postgres": "JSONB"},
            },
            {
                "name": "status",
                "field_type": "IntEnumFieldInstance",
                "db_column": "status",
                "python_type": "int",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": 1,
                "description": "on: 1\noff: 0",
                "docstring": None,
                "constraints": {"ge": -32768, "le": 32767},
                "db_field_types": {"": "SMALLINT"},
            },
        ],
        "fk_fields": [],
        "backward_fk_fields": [],
        "o2o_fields": [],
        "backward_o2o_fields": [],
        "m2m_fields": [],
    },
    "models.Email": {
        "name": "models.Email",
        "app": "models",
        "table": "email",
        "abstract": False,
        "description": None,
        "docstring": None,
        "unique_together": [],
        "indexes": [],
        "pk_field": {
            "name": "id",
            "field_type": "IntField",
            "db_column": "id",
            "python_type": "int",
            "generated": True,
            "nullable": False,
            "unique": True,
            "indexed": True,
            "default": None,
            "description": None,
            "docstring": None,
            "constraints": {"ge": 1, "le": 2147483647},
            "db_field_types": {"": "INT"},
        },
        "data_fields": [
            {
                "name": "email",
                "field_type": "CharField",
                "db_column": "email",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 200},
                "db_field_types": {"": "VARCHAR(200)"},
            },
            {
                "name": "is_primary",
                "field_type": "BooleanField",
                "db_column": "is_primary",
                "python_type": "bool",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": False,
                "description": None,
                "docstring": None,
                "constraints": {},
                "db_field_types": {"": "BOOL", "sqlite": "INT"},
            },
            {
                "name": "user_id",
                "field_type": "IntField",
                "db_column": "user_id",
                "python_type": "int",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"ge": 1, "le": 2147483647},
                "db_field_types": {"": "INT"},
            },
        ],
        "fk_fields": [
            {
                "name": "user",
                "field_type": "ForeignKeyFieldInstance",
                "python_type": "models.User",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {},
                "raw_field": "user_id",
                "on_delete": "CASCADE",
            }
        ],
        "backward_fk_fields": [],
        "o2o_fields": [],
        "backward_o2o_fields": [],
        "m2m_fields": [],
    },
    "models.Product": {
        "name": "models.Product",
        "app": "models",
        "table": "product",
        "abstract": False,
        "description": None,
        "docstring": None,
        "unique_together": [],
        "indexes": [],
        "pk_field": {
            "name": "id",
            "field_type": "IntField",
            "db_column": "id",
            "python_type": "int",
            "generated": True,
            "nullable": False,
            "unique": True,
            "indexed": True,
            "default": None,
            "description": None,
            "docstring": None,
            "constraints": {"ge": 1, "le": 2147483647},
            "db_field_types": {"": "INT"},
        },
        "data_fields": [
            {
                "name": "name",
                "field_type": "CharField",
                "db_column": "name",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 50},
                "db_field_types": {"": "VARCHAR(50)"},
            },
            {
                "name": "view_num",
                "field_type": "IntField",
                "db_column": "view_num",
                "python_type": "int",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": "View Num",
                "docstring": None,
                "constraints": {"ge": -2147483648, "le": 2147483647},
                "db_field_types": {"": "INT"},
            },
            {
                "name": "sort",
                "field_type": "IntField",
                "db_column": "sort",
                "python_type": "int",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"ge": -2147483648, "le": 2147483647},
                "db_field_types": {"": "INT"},
            },
            {
                "name": "is_reviewed",
                "field_type": "BooleanField",
                "db_column": "is_reviewed",
                "python_type": "bool",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": "Is Reviewed",
                "docstring": None,
                "constraints": {},
                "db_field_types": {"": "BOOL", "sqlite": "INT"},
            },
            {
                "name": "type",
                "field_type": "IntEnumFieldInstance",
                "db_column": "type_db_alias",
                "python_type": "int",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": "Product Type",
                "docstring": None,
                "constraints": {"ge": -32768, "le": 32767},
                "db_field_types": {"": "SMALLINT"},
            },
            {
                "name": "image",
                "field_type": "CharField",
                "db_column": "image",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 200},
                "db_field_types": {"": "VARCHAR(200)"},
            },
            {
                "name": "body",
                "field_type": "TextField",
                "db_column": "body",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {},
                "db_field_types": {"": "TEXT", "mysql": "LONGTEXT"},
            },
            {
                "name": "created_at",
                "field_type": "DatetimeField",
                "db_column": "created_at",
                "python_type": "datetime.datetime",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"readOnly": True},
                "db_field_types": {
                    "": "TIMESTAMP",
                    "mysql": "DATETIME(6)",
                    "postgres": "TIMESTAMPTZ",
                },
                "auto_now_add": True,
                "auto_now": False,
            },
        ],
        "fk_fields": [],
        "backward_fk_fields": [],
        "o2o_fields": [],
        "backward_o2o_fields": [],
        "m2m_fields": [
            {
                "name": "categories",
                "field_type": "ManyToManyFieldInstance",
                "python_type": "models.Category",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {},
                "model_name": "models.Category",
                "related_name": "products",
                "forward_key": "category_id",
                "backward_key": "product_id",
                "through": "product_category",
                "on_delete": "CASCADE",
                "_generated": False,
            }
        ],
    },
    "models.User": {
        "name": "models.User",
        "app": "models",
        "table": "user",
        "abstract": False,
        "description": None,
        "docstring": None,
        "unique_together": [],
        "indexes": [],
        "pk_field": {
            "name": "id",
            "field_type": "IntField",
            "db_column": "id",
            "python_type": "int",
            "generated": True,
            "nullable": False,
            "unique": True,
            "indexed": True,
            "default": None,
            "description": None,
            "docstring": None,
            "constraints": {"ge": 1, "le": 2147483647},
            "db_field_types": {"": "INT"},
        },
        "data_fields": [
            {
                "name": "username",
                "field_type": "CharField",
                "db_column": "username",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 20},
                "db_field_types": {"": "VARCHAR(20)"},
            },
            {
                "name": "password",
                "field_type": "CharField",
                "db_column": "password",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 200},
                "db_field_types": {"": "VARCHAR(200)"},
            },
            {
                "name": "last_login",
                "field_type": "DatetimeField",
                "db_column": "last_login",
                "python_type": "datetime.datetime",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": "<function None.now>",
                "description": "Last Login",
                "docstring": None,
                "constraints": {},
                "db_field_types": {
                    "": "TIMESTAMP",
                    "mysql": "DATETIME(6)",
                    "postgres": "TIMESTAMPTZ",
                },
                "auto_now_add": False,
                "auto_now": False,
            },
            {
                "name": "is_active",
                "field_type": "BooleanField",
                "db_column": "is_active",
                "python_type": "bool",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": True,
                "description": "Is Active",
                "docstring": None,
                "constraints": {},
                "db_field_types": {"": "BOOL", "sqlite": "INT"},
            },
            {
                "name": "is_superuser",
                "field_type": "BooleanField",
                "db_column": "is_superuser",
                "python_type": "bool",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": False,
                "description": "Is SuperUser",
                "docstring": None,
                "constraints": {},
                "db_field_types": {"": "BOOL", "sqlite": "INT"},
            },
            {
                "name": "avatar",
                "field_type": "CharField",
                "db_column": "avatar",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": "",
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 200},
                "db_field_types": {"": "VARCHAR(200)"},
            },
            {
                "name": "intro",
                "field_type": "TextField",
                "db_column": "intro",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": "",
                "description": None,
                "docstring": None,
                "constraints": {},
                "db_field_types": {"": "TEXT", "mysql": "LONGTEXT"},
            },
            {
                "name": "longitude",
                "unique": False,
                "default": None,
                "indexed": False,
                "nullable": False,
                "db_column": "longitude",
                "docstring": None,
                "generated": False,
                "field_type": "DecimalField",
                "constraints": {},
                "description": None,
                "python_type": "decimal.Decimal",
                "db_field_types": {"": "DECIMAL(12,9)", "sqlite": "VARCHAR(40)"},
            },
        ],
        "fk_fields": [],
        "backward_fk_fields": [
            {
                "name": "categorys",
                "field_type": "BackwardFKRelation",
                "python_type": "models.Category",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": "User",
                "docstring": None,
                "constraints": {},
            },
            {
                "name": "emails",
                "field_type": "BackwardFKRelation",
                "python_type": "models.Email",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {},
            },
        ],
        "o2o_fields": [],
        "backward_o2o_fields": [],
        "m2m_fields": [],
    },
    "models.Aerich": {
        "name": "models.Aerich",
        "app": "models",
        "table": "aerich",
        "abstract": False,
        "description": None,
        "docstring": None,
        "unique_together": [],
        "indexes": [],
        "pk_field": {
            "name": "id",
            "field_type": "IntField",
            "db_column": "id",
            "python_type": "int",
            "generated": True,
            "nullable": False,
            "unique": True,
            "indexed": True,
            "default": None,
            "description": None,
            "docstring": None,
            "constraints": {"ge": 1, "le": 2147483647},
            "db_field_types": {"": "INT"},
        },
        "data_fields": [
            {
                "name": "version",
                "field_type": "CharField",
                "db_column": "version",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 255},
                "db_field_types": {"": "VARCHAR(255)"},
            },
            {
                "name": "app",
                "field_type": "CharField",
                "db_column": "app",
                "python_type": "str",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {"max_length": 20},
                "db_field_types": {"": "VARCHAR(20)"},
            },
            {
                "name": "content",
                "field_type": "JSONField",
                "db_column": "content",
                "python_type": "Union[dict, list]",
                "generated": False,
                "nullable": False,
                "unique": False,
                "indexed": False,
                "default": None,
                "description": None,
                "docstring": None,
                "constraints": {},
                "db_field_types": {"": "TEXT", "postgres": "JSONB"},
            },
        ],
        "fk_fields": [],
        "backward_fk_fields": [],
        "o2o_fields": [],
        "backward_o2o_fields": [],
        "m2m_fields": [],
    },
}

gck123 · 2024-11-20T02:19:42Z

When the KeyError is fixed, there is another problem about the unique index deletion failure, this problem will only be reported by the database when the migration sql is executed, this problem is caused by the unique index name prefix idx_ and uid_ in the sql is messed up.

henadzit · 2024-11-23T12:01:41Z

@gck123 thank you for the snippet but it would be great if the PR updated the tests to include the test for this issue. Otherwise, it might come back as a regression.

Add unittest for drop unique field

waketzheng · 2024-12-02T09:44:38Z

@henadzit Cloud you review it?

Fix KeyError when deleting a field with unqiue=True

4e34158

gck123 closed this Nov 20, 2024

gck123 reopened this Nov 20, 2024

waketzheng and others added 4 commits November 29, 2024 16:15

Merge remote-tracking branch 'upstream/dev' into fix-drop-unique

0544a36

refactor: rename old_data_unique to is_unique_field

856b74d

Add testcases for remove unique field

14bc0e0

Merge pull request #1 from waketzheng/fix-drop-unique

c1b141d

Add unittest for drop unique field

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix KeyError when deleting a field with unqiue=True #365

Fix KeyError when deleting a field with unqiue=True #365

gck123 commented Nov 14, 2024 •

edited by waketzheng

Loading

henadzit commented Nov 18, 2024

gck123 commented Nov 20, 2024

gck123 commented Nov 20, 2024

henadzit commented Nov 23, 2024

waketzheng commented Dec 2, 2024

Fix KeyError when deleting a field with unqiue=True #365

Are you sure you want to change the base?

Fix KeyError when deleting a field with unqiue=True #365

Conversation

gck123 commented Nov 14, 2024 • edited by waketzheng Loading

henadzit commented Nov 18, 2024

gck123 commented Nov 20, 2024

gck123 commented Nov 20, 2024

henadzit commented Nov 23, 2024

waketzheng commented Dec 2, 2024

gck123 commented Nov 14, 2024 •

edited by waketzheng

Loading