-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Apply optimizations to queries to make use of adjacent ids #1961
Apply optimizations to queries to make use of adjacent ids #1961
Conversation
de39e9c
to
a9c91c5
Compare
a9c91c5
to
fd0e777
Compare
@rngcntr Thank you very much for new optimizations! Could you please rebase your PR against the latest master commit to resolve conflicts? |
@porunov I will do so as soon as I'm finally able to make all tests pass |
c963c0f
to
aa674d8
Compare
So I've been working on this and there still remains one case which results in a flaky test. As I don't think I'm able to solve this problem myself, I'm looking for help here. janusgraph/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphTest.java Lines 4184 to 4192 in aa674d8
Because this traversal matches one of the given patterns, when applied, my strategy should rewrite it to look like this: janusgraph/janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphTest.java Lines 4175 to 4183 in aa674d8
And as expected, it does so deterministically. But when applying all optimization strategies one after another, some of the built-in TinkerPop strategies cause both query plans to diverge:
The differences are introduced by two strategies.
I found out the presence of I'm currently evaluating why |
0c63a36
to
8d5093e
Compare
Update: We tracked the inconsistencies down to a TinkerPop Bug None of the failed tests were directly caused by the newly implemented optimization strategies but yet they have randomly led the way to discover this bug. I have rewritten the affected queries to avoid this bug in the tests and as far as I can tell, all tests are passing now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @rngcntr ! Looks good to me. I've checked the tests, they are failing without optimizations as expected.
I have only several nitpick comments. Otherwise looks very good!
...src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexOptimizerStrategy.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexOptimizerStrategy.java
Show resolved
Hide resolved
...src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexOptimizerStrategy.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexOptimizerStrategy.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexOptimizerStrategy.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexOptimizerStrategy.java
Outdated
Show resolved
Hide resolved
...ain/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexHasIdOptimizerStrategy.java
Outdated
Show resolved
Hide resolved
janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphTest.java
Outdated
Show resolved
Hide resolved
8d5093e
to
422a840
Compare
I see my intensive search for bugs has left it's traces in the code :D |
Signed-off-by: Florian Grieskamp <[email protected]>
422a840
to
0f7af8d
Compare
Here is an overview of what got changed by this pull request: Complexity increasing per file
==============================
- janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexOptimizerStrategy.java 7
- janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexHasIdOptimizerStrategy.java 3
- janusgraph-core/src/main/java/org/janusgraph/graphdb/tinkerpop/optimize/AdjacentVertexIsOptimizerStrategy.java 2
See the complete overview on Codacy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thank you @rngcntr !
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All TP tests passed for BerkeleyJanusGraphProcessTest
Only one failed org.apache.tinkerpop.gremlin.process.traversal.step.map.ProfileTest#testProfileStrategyCallback
but on master too
@mad Last time I was executing TP tests during the update of TinkerPop to 3.4.6 version. During that time all TP tests passed. This means that we introduced a bug later after that update. We need to investigate where we introduced that problem to fix it. |
This feature was originally part of #1937 which was splitted to decrease per-PR-complexity.
This PR introduces two new OptimizerStrategies, which further improve the existing optimizations based on adjacent vertex ids. In the current state, these queries will be optimized to check the adjacency on the edge before iterating to the vertex itself:
g.V(left).outE('label').where(__.inV().is(right))
g.V(left).outE('label').filter(__.inV().is(right))
The optimization of these queries is within the scope of #1960:
g.V(left).outE('label').where(__.inV().hasId(right))
g.V(left).outE('label').filter(__.inV().hasId(right))
With the new optimizer, these queries will also profit of optimization:
g.V(left).outE('label').inV().is(right)
g.V(left).outE('label').inV().hasId(right.id())
g.V(left).out('label').is(right)
g.V(left).out('label').hasId(right.id())
g.V(left).out('label').filter(is(right))
g.V(left).out('label').where(is(right))
g.V(left).out('label').filter(hasId(right.id()))
g.V(left).out('label').where(hasId(right.id()))
All of this also applies to all other valid variations of
in()/out()/both()
.For all changes:
master
)?For code changes: