hello from Intake

[martindurant]

I was pointed to your project by a colleague who saw your blog post in some news stream.

Let me introduce you to Intake https://intake.readthedocs.io/en/latest/

We seem to have a lot of overlapping interests and functionality. Intake has been around for some time and already supports many of the things you mention in your blog post:

• a simply python library which only does cataloging
• many data formats and limited in-memory data types (dataframe, array, itarable, xarray, stream)
• support for reading over any remote backend supported by fsspec (http, s3, gcs, azure, ftp, gdrive...)
• a REST server and protocol
• a CLI
• a simply python library which only does cataloging and load and then gets out of the way
• crawlers for remote data services with catalogs of their own like SQL
• a simple YAML-based catalog format for building your own
• streaming data
• plugin systems for many of the components

Rather than reinventing all these things here, I mildly suggest you might consider making use of the ones already existing. I haven't yet got my head around what differentiates recap from Intake - do you think you can summarize? Maybe we can learn something from you, or we can collaborate to make something better for everyone.

[criccomini]

Neat! I was not familiar with intake. It does, indeed, share some characteristics. I'll have to dig into it in more detail. It strikes me that we have similar software but slightly different focuses; intake seems more data science focused than Recap, for example.

One of the lovely things about open source is that it allows for many similar things to grow and flourish. :)

[martindurant]

Certainly agree on the latter point, and my job relies on it :)

I would say that Intake's focus is on "data" in general, but a lot of the numerical stack in python is data-science oriented, or at least related. What would you say is Recap's focus?

I was not familiar with intake.

It saddens me somewhat. If you agree that Intake is useful for at least some population of data people, then maybe you have some suggestions for how to make it more prominent. It need not be the only catalog solution, but it certainly is stable and has some pedigree by now.