Move files from /arriving/ to /archived/ at shutdown

[Fokko]

Hi all,

Right now, when we shut down Divolte because of scaling-down or in case of a rolling update, the sinks are closed and flushed. One of the problems is that the files remain in the /arriving/ directory. We can solve this in two ways:

• When shutting down Divolte, flush the buffers and move the files to the right directory
• Write an Airflow job that will clean up these files, check if they are non-empty and move them to the right directory.

The latter is more secure I'd say, in case Divolte crashes of gets killed for an arbitrary reason, the file will stay there.

Any thoughts?

Cheers, Fokko

[friso]

If files are closed cleanly during shutdown, moving them as part of shutdown makes sense to me.

Moving stray files to the publish location without inspection seems dangerous to me. There are no guarantees about their validity, integrity, etc.

[asnare]

I'm a little surprised they're not being published as part of a clean shutdown. I'd call that a bug.

[Fokko]

This instance did shutdown cleanly:

Unfortuanlly we still have files in arriving:

  12991802  2018-04-11T23:10:40Z  gs://xxx/arriving/20180411230809-divolte-tracking-divolte-7bb54f6c9b-pv2bk-1.avro
   1322276  2018-04-11T23:10:40Z  gs://xxx/arriving/20180411230809-divolte-tracking-divolte-7bb54f6c9b-pv2bk-1.avro.part

I'll look into this the coming days.