Attach precise release timestamp to kafka messages

[theferrit32]

Could attempt to look up release file on clinvar ftp site. The problem is that within our system, we might have records modified between the time clinvar releases and it makes it to our ingest, resulting in out of order changes.

Add field release_timestamp with precise ISO8601 datetime of when original upstream release message was received, so all messages within the release have the same release_date and same release_timestamp. In this case downstream records timestamped between release_timestamp and the receipt of the messages created here will be "overridden" and flagged for review. Chance of this scenario is relatively small.

[theferrit32]

@larrybabb if this is something we still need we should see whether we or dsp can pull this precise value from the source FTP file. Or we should think about whether/why this is necessary.

[theferrit32]

I think this should be bumped up in priority just so that we don't let it go too far without a more sound handling of timestamps and comparison logic in downstream applications (genegraph)

[theferrit32]

SPARQL 1.1 supports logic operators for xsd:dateTime objects. If we use a consistent fully-qualified ISO8601 formatting with +00:00 timezone, we can also use simple string comparison to determine relative ordering (until the year 10000). Right now we are not storing timestamps in records in genegraph as xsd:dateTime.

https://www.w3.org/TR/sparql11-query/#OperatorMapping

[larrybabb]

i'll defer to @tnavatar and @toneillbroad for the final say