percolate option on index and bulk

kimchy · kimchy · commit 033426f85233 · 2011-02-17T23:17:08.000+02:00
diff --git a/guide/reference/api/bulk.textile b/guide/reference/api/bulk.textile
@@ -48,6 +48,10 @@ h1. Routing
 
 p. Each bulk item can include the routing value using the @_routing@ field. It automatically follows the behavior of the index / delete operation based on the @_routing@ mapping.
 
+h1. Percolator
+
+p. Each index bulk item can include a percolate value using the @_percolate@ field.
+
 h1. Parent
 
 p. Each bulk item can include the parent value using the @_parent@ field. It automatically follows the behavior of the index / delete operation based on the @_parent@ / @_routing@ mapping.
diff --git a/guide/reference/api/index_.textile b/guide/reference/api/index_.textile
@@ -128,6 +128,27 @@ $ curl -XPUT localhost:9200/blogs/blog_tag/1122?parent=1111 -d '{
 }'
 </pre>
 
+h1. Percolate
+
+p. "Percolation":percolate.html can be automatically done on an indexed doc by passing the @percolate@ parameter. Setting it to @*@ will cause all percolation queries registered against the index to be checked against the indexed doc, for example:
+
+<pre class="prettyprint">
+curl -XPUT localhost:9200/test/type1/1?percolate=* -d '{
+    "field1" : "value1"
+}'
+</pre>
+
+p. It can also be set to query (following the query string syntax) to filter out which percolator queries will be executed:
+
+<pre class="prettyprint">
+curl -XPUT localhost:9200/test/type1/1?percolate=color:green -d '{
+    "field1" : "value1",
+    "field2" : "value2"
+}'
+</pre>
+
+p. Percolation on index operation is done while optimizing the distributed nature of elasticsearch. Once the index operation is done on the primary shard, it is sent to all the replicas, and while the operation is done on the replicas, the percolation is executed on the node hosting the primary shard. Also, the parsing operation done on the primary shard is reused for the percolation operation.
+
 h1. Distributed
 
 p. The index operation gets hashed into a specific shard id. It then gets redirected into the primary shard within that id group, and replicated (if needed) to shard replicas within that id group.
diff --git a/guide/reference/api/percolator.textile b/guide/reference/api/percolator.textile
@@ -0,0 +1,88 @@
+---
+layout: guide
+title: Percolator API
+cat: guide
+sidebar: reference_api
+---
+
+p. The percolator allows to register queries against an index, and then send @percolate@ requests which include a doc, and getting back the queries that match on that doc out of the set of registered queries.
+
+p. Think of it as the reverse operation of indexing and then searching. Instead of sending docs, indexing them, and then running queries. One sends queries, registers them, and then sends docs and finds out which queries match that doc.
+
+p. As an example, a user can register an interest (a query) on all tweets that contain the word "elasticsearch". For every tweet, one can percolate the tweet against all registered user queries, and find out which ones matched.
+
+p. Here is a quick sample, first, lets create a @test@ index:
+
+<pre class="prettyprint">
+curl -XPUT localhost:9200/test
+</pre>
+
+p. Next, we will register a percolator query with a specific name called @kuku@ against the @test@ index:
+
+<pre class="prettyprint">
+curl -XPUT localhost:9200/_percolator/test/kuku -d '{
+    "query" : {
+        "term" : {
+            "field1" : "value1"
+        }
+    }
+}'
+</pre>
+
+p. And now, we can percolate a document and see which queries match on it (note, its not really indexed!):
+
+<pre class="prettyprint">
+curl -XGET localhost:9200/test/type1/_percolate -d '{
+    "doc" : {
+        "field1" : "value1"
+    }
+}'
+</pre>
+
+p. And the matches are part of the response:
+
+<pre class="prettyprint">
+{"ok":true, "matches":["kuku"]}
+</pre>
+
+h1. Filtering Executed Queries
+
+p. Since the registered percolator queries are just docs in an index, one can filter the queries that will be used to percolate a doc. For example, we can add a @color@ field to the registered query:
+
+<pre class="prettyprint">
+curl -XPUT localhost:9200/_percolator/test/kuku -d '{
+    "color" : "blue"
+    "query" : {
+        "term" : {
+            "field1" : "value1"
+        }
+    }
+}'
+</pre>
+
+p. And then, we can percolate a doc that only matches on blue colors:
+
+<pre class="prettyprint">
+curl -XGET localhost:9200/test/type1/_percolate -d '{
+    "doc" : {
+        "field1" : "value1"
+    },
+    "query" : {
+        "term" : {
+            "color" : "blue"
+        }
+    }
+}'
+</pre>
+
+h1. How it Works
+
+p. The @_percolator@ which holds the repository of registered queries is just a another index. The query is registered under a concrete index that exists (or will exist). That index name is represented as the type in the @_percolator@ index (a bit confusing, I know...).
+
+p. The fact that the queries are stored as docs in another index (@_percolator@) gives us both the persistency nature of it, and the ability to filter out queries to execute using another query.
+
+p. The @_percolator@ index uses the @index.auto_expand_replica@ setting to make sure that each data node will have access locally to the registered queries, allowing for fast query executing to filter out queries to run against a percolated doc.
+
+p. The percolate API uses the whole number of shards as percolating processing "engines", both primaries and replicas. In our above case, if the @test@ index has 2 shards with 1 replica, 4 shards will round robing in handing percolate requests. (dynamically) increasing the number of replicas will increase the number of percolation power.
+
+p. Note, percolate request will prefer to be executed locally, and will not try and round robin across shards if a shard exists locally on a node that received a request (for example, from HTTP). Its important to do some roundrobin in the client code among nodes (in any case its recommended). If this behavior is not desired, the @prefer_local@ parameter can be set to @false@ to disable it.
