-
Notifications
You must be signed in to change notification settings - Fork 148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
no rows in Trino when pyspark xtable source from delta and write into Hudi and Iceberg with S3 + HMS #460
Comments
running xtable again
running sync again
|
Hey @alberttwong, do you have the hudi connector present in the trino version you are running ? If possible can you share the trino environment details you are using ? https://xtable.apache.org/docs/trino |
BTW, Trino works fine with source hudi and target iceberg and delta. |
@alberttwong is there a way to pass in reader options to Trino for reading hoodie? |
I'll give it a try. What is weird is that this option isn't listed at https://trino.io/docs/current/connector/hudi.html |
won't work in Trino 428
|
trinodb/trino#15368 implies that the hudi setting in hudi.properties should be also trino 428 shows there isn't any metadata like property I can set. https://github.com/trinodb/trino/blob/428/plugin/trino-hudi/src/main/java/io/trino/plugin/hudi/HudiConfig.java Also I don't know how to upgrade the hudi connector in Trino. There are no instructions.
|
looking up Trino 428, they're using Hudi 0.12.3 https://github.com/trinodb/trino/blob/428/plugin/trino-hudi/pom.xml. Trino 440 is the first one to use Hudi 0.14.1 https://github.com/trinodb/trino/blob/440/plugin/trino-hudi/pom.xml. Do you want me to upgrade my image to at least Trino 440? |
I just tried out Trino 449 and it has the same error
|
I'll retry writing delta and xtable and hudi-hive-sync/run_sync_tool.sh with Trino 449 tomorrow morning. |
confirmed that upgrading to Trino 449 doesn't work. It looks like we need to make changes to the Trino Hudi Plugin. |
linked Trino github issue. trinodb/trino#22289 |
hmm... another wrinkle. apache/hudi#10289 |
I was told to downgrade to Trino 418. |
Downgrade doesn't work.
|
Comment from @yihua Trino 418 depends on Hudi 0.12.3 while Iceberg table with Hudi metadata requires Hudi 0.14.0+ so Trino 418 does not work in your use case. |
Search before asking
Please describe the bug 🐞
I wrote 6 rows into delta on s3 and then used xtable to covert the data to iceberg and hudi. Next I used run_sync_tool.sh to register the hudi files in HMS and then query it into Trino. No data in Trino.
I don't see any errors.
xtable looks good
hudi-hive-sync/run_sync_tool.sh looks good
trino should show rows
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: