Loader: Add legacy file hierarchy support #1320
Conversation
oguzhanunlu
force-pushed
the
loader-legacy-support
branch
from
b39b210
to
5dba819
Compare
oguzhanunlu
changed the title
| ): LoadStatements = { | ||
| val shreddedStatements = discovery.shreddedTypes | ||
| .filterNot(_.isAtomic) | ||
| .groupBy(_.getLoadPath) | ||
| .groupBy(_.getLoadPath(legacyPartitioning)) |
There was a problem hiding this comment.
I think lines 97-101 are now incorrect for the case when legacy partitioning is enabled and if the schema requires a recovery table.
If legacy partitioning is disabled, then it is OK. And if schema evolution rules are not broken, then it is OK.
Imagine this batch uses versions 1-0-0 and 1-0-1 of the same schema. And imagine 1-0-1 is broken so needs a recovery table.
When you call .map(_.head) you do not know whether the head is 1-0-0 or 1-0-1. So when you call mergeResult = .... (line 97) you don't know if you get the good schema or the recovery schema.
Before I offer a solution... I need to understand the corresponding feature in the transformer, #1315 . I have not yet read through that PR, so shall do so next.
|
This is cancelled |
No description provided.