Transformer: Support old output file hierarchy

[oguzhanunlu]

No description provided.

[spenes]

LGTM!

[istreeter]

Consider a pair of schemas 1-0-0 and 1-0-1, but imagine schema evolution rules are broken. For example:

1-0-0

{
  "a": {"type": "string"}
}

1-0-1

{
  "a": {"type": "integer"},
  "b": {"type": "string"}
}

If we have legacyPartitioning = false (i.e. the new default behaviour) then everything is OK. The TSV files written to the 1-0-0 directory will have a column for field a, and the TSV files written to the 1-0-1 directory will have columns for fields a and b. The loader knows how to load the 1-0-0 directory into the main *_1 table, and how to load the 1-0-1 directory into the recovery table.

But, if we have legacyPartitioning = true, then the output directory contains a mixture of TSV files with different number of columns. Some TSV files will have a column for field b and some will not.

@oguzhanunlu please could you follow through my reasoning and check I have got that correct??

Bear in mind that a directory containing mixed TSV files is very unhelpful for any purpose. It certainly cannot be loaded into a warehouse table, and it is not helpful for any other purpose either.

Possible solutions

We could say that the legacyPartitioning flag must only be used on pipelines that have no broken schemas. This is the very least we need to do.

But what if we enable it on a pipeline and then six months later there is a broken schema?

The best solution I can think of right now is this (when legacyPartitioning is true):

1. If shredded output used the merged schema, then write to the legacy /model=1 directory.
2. If shredded output used a recovery schema, then write to somewhere underneath /type=recovered, instead of type=good.