-
Notifications
You must be signed in to change notification settings - Fork 83
Elastic Search Store #205
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Elastic Search Store #205
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you add type on all public vals/defs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
very cool. Glad to see twttr is picking up on json4s. I'm a fan of a common (ast) interface that you can swap out backends for :)
|
@johnynek I will incorporate the feedback on injections in bijection project as there is already a PR there. I may not need the monoid in the end, if thats the case I will remove them |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should nodes be a List[(String, Int)]?
Still learning about elasticsearch architecture. I'm wondering if there are ever cases where two instances run on the same host.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about moving this to storehaus-core? There can possibly be a mysql version of this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am fine with that as long as @johnynek is cool with that. I suspect he will ask me to change some things around. In addition to MySql we can potentially use QueryableStore for HBase and Mongo as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you make this: QueryableStore[-Q, +V]? I don't see the need to force this to be a subclass of Store[K, V] (for instance, I can't write to Google, but I can query it).
Also, and I think Avi asked this, what is the main difference here to: Readable[Q, Seq[V]]
Lastly, if we are in the business of making new traits, why not make this: query(q: Q): Future[Seq[V]]?
Whatever we decide here should definitely be in storehaus-core.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My preference is to add a method to do something like:
def queryable: Readable[Q, Seq[V]]
longer term, getting something like the Spool in the value might be the right thing, but I am skeptical of large async requests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this safe to call from different threads? Link to docs allowing that?
|
@johnynek I have incorporated your feedback, please take a look. I have gone to some length to make sure that when you convert a Store, QueryableStore gets converted as well. Let me know if it make sense |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a thought - if we are going to use ReadableStore, can we just make this:
trait QueryableStore[Q, V] extends ReadableStore[Q, Seq[V]]There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm I am not sure about that. To me QueryableStore produces a ReadableStore but is not a ReadableStore itself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok makes sense. I can see the goal for having this trait as a kind of factory class -- enabling a store to produce a queryable version of itself.
(Was just wondering if there is a way to avoid a layer of indirection here if we thought of a queryable store as a ReadableStore that represents a map from Query to Seq[V]. But I guess there's no good way to achieve the above goal if we go down this path.)
Alternatively, can there be a combinator for this?
class QueryableStore[Q, K, V](store: ReadableStore[K, V])
extends ReadableStore[Q, Seq[V])Sorry, just trying to be useful :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's make it covariant to match store.
trait QueryableStore[Q, +V]
Q could possibly be made contra variant, but it a bit of a pain due to the way we are returning a store. Also, queries are less likely to be subclassed. I'd leave that as is.
|
Tested with SummingBird and had to change some code to handle failed queries. Everything is working fine at this point and I am able to write and read from Elastic Search from SummingBird |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since the bijection code got merged, can we make this private so we don't get a binary incompatibility when we remove it?
I want to publish this week. That or we can get bijection out today, and then fix this up tomorrow or thurs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you can push bijection out today, I can make the change by tomorrow. Let
me know if this works
On Tue, Feb 4, 2014 at 1:54 PM, P. Oscar Boykin [email protected]:
In
storehaus-elasticsearch/src/main/scala/com/twitter/storehaus/elasticsearch/Injections.scala:+import scala.util.Try
+import org.json4s._
+import native.JsonMethods._
+import com.twitter.bijection.Inversion.attempt
+import native.Serialization.{write, read}
+import com.twitter.algebird.Monoid
+import org.json4s.JValue
+import com.twitter.algebird.
bijection.BijectedMonoid
+
+
+/**since the bijection code got merged, can we make this private so we don't
get a binary incompatibility when we remove it?I want to publish this week. That or we can get bijection out today, and
then fix this up tomorrow or thurs?Reply to this email directly or view it on GitHubhttps://github.com//pull/205/files#r9435670
.
|
@johnynek I have upgraded to bijection 0.6.2. Build is failing as storehaus-cache tests failed, not sure why that is, maybe retrigger the build? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these whitespace/comment only changes are not ideal. Please try to keep the diff as focused as possible on the functional bits. This makes it easier to miss meat in a code review.
|
ready to merge with a few minor cleanups as noted above. Thanks for patiently working on this. |
|
@johnynek made appropriate changes, take a look now |
|
Great to meet you Tuesday! Two more minor changes and it's in! |
Task List: