ESQL: Lookup Join meta issue

[costin]

Description

Meta ticket around the main implementation topics required for lookup join

• add grammar / parser LOOKUP JOIN
• Make it execute correctly on data nodes, too, and enable csv tests ...OnTheDataNode. See (reproducible) failed runs from ESQL: Fix lookup join output #117639 branch:
  • https://gradle-enterprise.elastic.co/s/5cihyuwehu2lu
  • https://gradle-enterprise.elastic.co/s/aox7wtnevoij2
  • Enhance LOOKUP JOIN csv-spec tests to cover more cases and fix several bugs found #117843
• USING condition
  • rewrite into JOIN equi-join
  • make ON field alias to USING field (syntax already used in enrich) - alternative is to deprecate this syntax
  • change index name grammar for identifier to source (to allow date math and others)
  • enhance the field matching grammar against various declarations such as parent.subfield & co.
• CSV tests (some suggestions to start with: Esql/lookup join grammar #116515 (comment))
  • Cover primary use cases: Enhance LOOKUP JOIN csv-spec tests to cover more cases and fix several bugs found #117843
• Known bugs
  • Make PushDownAndCombineLimits work for Joins ESQL: LookupJoin fails to push down limits #117698
  • Additional null rows ESQL: LOOKUP JOIN produces additional null rows #117702
  • EsqlSession.fieldNames does not handle lookup references that are also mentioned in aliases (erases them)
  • EsqlSession calls preAnalyzeLookupIndices asking for all the lookup index' fields all the time, c.f. Enhance LOOKUP JOIN csv-spec tests to cover more cases and fix several bugs found #117843 (comment)
  • ReplaceMissingFieldWithNull rule does not have access to SearchStats for lookup indices
• name qualifiers (for disambiguation) (look to descope)
• field resolution
  • establish shadowing rules
  • clarify in USING/ON pattern the rules around the join key which are following downstream (left vs right)
  • make the other side fields nullable (this changes the equality for existing fields causing side-effects) create new fields or update the entire field declaration.
  • check resolution exception to differentiate between 500 and 400 (Esql/lookup join grammar #116515 (comment))
• clarify output order rules in case of common keys (join keys and columns with same names) for consistency
  • SQL JOIN:
    • columns are kept in place and in lacking qualifiers, lead to ambiguation errors
    • JOIN USING moves the join keys in front followed by the columns in table declaration order
  • in ESQL the columns override each other:
    • in EVAL, the latest declaration wins alongside its position - all previous declarations are hidden as if they were removed
    • existing LOOKUP and INLINESTATS return left columns, join keys, right columns using EVAL rules for non-joining keys
    • STATS BY x, returns grouping keys last (declaration order)
  • proposal:
    • a. return join keys first (like SQL first)
    • b. return join keys last (declaration order, similar to STATS BY - clashes with USING)
  • c.f. output computation in Analyzer (Esql/lookup join grammar #116515 (comment))
• Look whether the LookupJoin output could be computed inside the class ESQL: Fix lookup join output #117639
• Resolve lookup table mappings - LOOKUP JOIN using field-caps for field mapping #117246
  • Update lookup table mappings solution to support multiple join commands - Craig
• Update verifier and add corresponding tests Esql/lookup join grammar #116515 (comment).
  • the existing CSV tests are very basic (one specifically filters out all entries) and needs a lot more beefing
  • disallow lookup joins on remote clusters/validate against CCQ
• Enhance tests
  • Cover primary use cases: Enhance LOOKUP JOIN csv-spec tests to cover more cases and fix several bugs found #117843
  • Add negative tests (missing indexes, indexes with wrong mode, unresolvable field references, etc.)

Compute

• Make field extractor work with LookupOperators (keep field extraction in one place)
• Bifurcate join operator/allow other operators between join and Lookup source (eval, filter, etc..)
• Align LookupJoin and LookupJoinExec - the first allows branching, the second does not.
• Some kind of scale test

Maybe

• obtain IndexMode inside IndexResolver (FieldCaps)
  Later
• refactor JoinConfig into general Expression

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)