Dagster database bug

[lokinnnnnn]

What's the issue?

When Dagster initializes, the database will create a table called snapshots with a data type of blob. When the value stored in snapshot-body exceeds the size of the blob range, a series of cascading error reactions will occur. In the UI, clicking on 'view job' will display the title 'Unexpected GraphQL error'. The detailed content is as follows:

The backend log will display:

What did you expect to happen?

At that time, I thought there was a problem with the Python library version, so I tried changing the version of each component dozens of times, but the problem still couldn't be solved.
So I started with the warning message, which prompted: Warning: root: get needle snapshot: could not degrade bytes stored in snapshot table
Then I looked at my MySQL database and found a snapshot table, so I started researching how to parse snapshots with dagster. Later, I looked at the source code of dagster and used it to parse the snapshots table. My demo code gave me an error message, and the final problem was found to be insufficient field size. Finally, I manually changed the field type to medium blob. Problem solved.

How to reproduce?

1. Dagster needs to create a database in MySQL without changing any structure of the database.
2. We need a DBT project that includes many models to reach the upper limit of the blob.

Dagster version

1.8.7

Deployment type

Docker Compose

Deployment details

Package an image using Docker, upload it to AWS ECR, deploy grpc, daemon, and web server on AWS ECS, and use Fargate's startup type.

requirements.txt
dagster==1.9.0
dagster-graphql==1.9.0
dagster-webserver==1.9.0
dagster-aws==0.25.0
dagster_dbt==0.25.0
dagster-mysql==0.25.0
dbt-core==1.7.18
dbt-mysql==1.7.0

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
By submitting this issue, you agree to follow Dagster's Code of Conduct.