Neo4J Alternatives

[rstyd]

We're currently using an older version of neo4j due to some issues we have with newer versions. This is kind of risky because we're going to reach a time (might already have) where the older version we're using is no longer supported and things could just break down the line. We need to either find a working solution with the newer versions of neo4j or look at pivoting to another graph database.

I can't remember what all the issues were with moving to a newer version of neo4j. @Boogie3D Could you summarize those below? The primary difficulty in moving to another GDB is that Steven has written a massive amount of cypher code which is neo4j's proprietary query language which locks us in.

Fortunately, neo4j has been working on open sourcing cypher and there are several competing GDBs with cypher support. These include redis graph, agensgraph, and memgraph. I propose we take some time to check out these alternatives. One nice thing about each of these is they can all support multiple GDBs running in the same instance which would make life much easier for our workflow management backend.

[Boogie3D]

Pros to upgrading to Neo4j 4.x:

• Single-instance multi-database support
• Upgraded bolt protocol
  • Supports keepalive signal
  • New neo4j:// protocol includes bolt and routing protocols
• Increased security, user/role management

Cons:

• Complicates BEE packaging/distribution if upgraded post-release
• Cannot downgrade back to 3.5 easily if already deployed to users
• Several features still locked behind Enterprise edition
  • Requires paid subscription except under certain circumstances
  • Enterprise follows "open core" model, source code not freely available anywhere
  • Community edition is still open source under GPLv3
• Major changes to Python interface completely breaks the Neo4jDriver component

More information on Neo4j 4.0 changes and version migration here:

• https://neo4j.com/release-notes/database/neo4j-4-0-0/
• https://neo4j.com/docs/upgrade-migration-guide/current/

[pagrubel]

redis graph, asengraph, and memgraph have versions that use the open source version of cypher so the code we have will work.

[jmsuhy]

You can also use the DozerDB plugin to add enterprise features to Neo4j Community such as Multi-Database functionality. It's all vanilla GPL licensed. https://github.com/dozerdb/ and https://dozerdb.org

[pagrubel]

Meeting notes Jan 23:
Can illuminate: redis-graph

Most likely the two options are memgraph or going to the new version of neo4j or using RDF interface
RDF query language is sparql so would require rewriting cypher

[rstyd]

Here are the various options: https://opencypher.org/projects/.

[aquan9]

Here is my ArcadeDB Overview:

Documentation:

Pros:
Multi-model database - If we need a place to put metadata about the tasks and jobs being submitted we wouldn't need to launch a separate database component to be able to do that.
Supports Cypher
Supports other query languages like GraphQL if we ever decided to move away from Cypher

Cons:
Doesn't support Neo4j's BOLT interface
Cypher support seems not well maintained:

ArcadeDB partially supports Open Cypher as query engine.
It uses Cypher for Gremlin
under-the-hood which has stopped being supported a few years ago.
There are a lot of things that work and a lot of nuances which don’t.

ArcadeDB also doesn’t support Neo4j’s BOLT protocol. This means you can’t use a Neo4J driver with ArcadeDB server.
Known Limitations with the Cypher Implementation

ArcadeDB automatically handles the conversion between compatible types, such as strings and numbers when possible. Cypher does not. So if you define a schema with the ArcadeDB API and then you use Cypher for a traversal, ensure you’re using the same type you defined in the schema. For example, if you define a property "id" to be a string, and then you’re executing traversal by using integers for the ids, the result could be unpredictable.

ArcadeDB’s Cypher implementation is based on the Cypher For Gremlin Open Source transpiler. This project is not actively maintained by Open Cypher anymore, so issues in the transpiler are hard to fix. Please bear this in mind if you’re moving a large project in Cypher into ArcadeDB. The best way to address such issues is to rewrite the faulty cypher query into ArcadeDB SELECT or MATCH statement.

Potential Performance issues at scale

Consider that Cypher queries are translated into Gremlin. As much as ArcadeDB’s Gremlin implementation is optimized, queries run slower than using native SQL or Java API. Based on some internal benchmarks, ArcadeDB’s native SQL (SELECT or MATCH) is much faster than Cypher (even 2,000% faster!). The difference is larger with complex queries that work on many records. A simple Cypher MATCH with a lookup with a simple condition will be closer to native SQL performance, but a scan or huge traversal will be much slower with Cypher, because of the underlying usage of Gremlin is not optimized for extreme performance.

[pagrubel]

Notes from Jan 30 meeting:
Using a db for each workflow could really become a memory problem esp. if we have thousands of workflows.
We've thought of each workflow being a sub graph.
Currently: We start up a neo4j for each wofkflow and kill it when the workflow is done
Making a subgraph we could then have the database running whenever beeflow is running (move neo4j to core)
Gui uses wf manager not neo4j
Memgraph uses Cypher & Bolt
Neo4j free version doesn't support multiple graphs
Reads and Writes are transactional (already atomic)
If Memgraph supports multiple workflows and arm (containers for X86 and Mac M1) we should switch; if not stick with Neo4j - Rusty will look into this.
Visualization of Neo4j is better supported

Goal update to a more current version.
We also need to look at changing password mechanism. We could generate a password in the bee_workdir on the first start.

[jmsuhy]

Neo4j Community free version does support multiple graphs if you use the DozerDB Plugin. https://dozerdb.org