[DON'T MERGE] Proof of Concept: ES|QL approximate query execution #131828
Conversation
4345891 to
ee5caf5
Compare
| * | ||
| * To be so, the plan must contain at least one STATS function, and all | ||
| * functions between the source and the leftmost STATS function must be | ||
| * swappable with STATS. |
There was a problem hiding this comment.
You mean with SAMPLE here, right?
| * swappable with STATS. | |
| * swappable with SAMPLE. |
There was a problem hiding this comment.
yes, this should be SAMPLE
| * off at the leftmost STATS function, followed by "| STATS COUNT(*)". | ||
| * This value can be used to pick a good sample probability. | ||
| */ | ||
| public LogicalPlan countPlan() { |
There was a problem hiding this comment.
This extra query is probably my major "concern". It looks ok, but it's still going to execute evals and wheres, which could end up executing a full query anyway (?). It looks a bit "dangerous".
As an idea, I wonder if we could use some kind of Lucene statistics for this. I don't know if we have them though, or if what we have is enough. Even if they were just approximates, they could let us avoid this extra query, maybe. This would be another block of work though
There was a problem hiding this comment.
I get your concern. That's exactly why I wanted some early feedback.
The extra query is pretty similar to the extra query of the inline join subplan though.
There was a problem hiding this comment.
In the case of
FROM data | STATS COUNT()
I guess we can get the count directly from Lucene.
But for a more complicated
FROM data | WHERE my_function(x) < 1 | STATS COUNT()
that's obv not possible.
We can use sampling again though to get an approximate count, which is good enough for setting the probability.
| public LogicalPlan countPlan() { | ||
| Holder<Boolean> encounteredStats = new Holder<>(false); | ||
| LogicalPlan countPlan = logicalPlan.transformUp(plan -> { | ||
| if (plan instanceof LeafPlan) { |
There was a problem hiding this comment.
A Join may have 2 leaf plans if I'm not wrong, which could lead to this detecting the wrong Stats (?)
There was a problem hiding this comment.
I didn't give too too much thought yet on JOIN, FORK, INLINESTATS, etc.
First wanted to make something work end to end.
ee5caf5 to
ceaa209
Compare
Proof of concept for approximate query execution
This is for gathering early feedback; not for merging!
This is targeting queries of the form
Approximating rewrites it to
The sample probability is such that the approximated results are based on ~1000 docs. It's determined via the total result count:
You can use this as follows
With
"approximate": false, the (correct) results are:(based on
"documents_found": 4675)With
"approximate": true, the (approxmiate) results are like:(based on
"documents_found": 990)