[manifold] shard the partitioner #4
Conversation
lib/manifold.ex
Outdated
| partitioner(:erlang.phash2(pid, 4)) | ||
| end | ||
|
|
||
| defp partitioner(0), do: Manifold.Partitioner |
There was a problem hiding this comment.
this should be compile time generated. but i dont want to pull out the ol' macro handbook.
There was a problem hiding this comment.
When you add a macro its worth commenting why the first one has no number suffix. (backwards compat)
There was a problem hiding this comment.
This would be a pretty simple macro. What precisely do you want it to generate?
lib/manifold.ex
Outdated
| end | ||
|
|
||
| def partitioner_for(pid) do | ||
| partitioner(:erlang.phash2(pid, 4)) |
There was a problem hiding this comment.
any reference of 3, or 4, should be a config.
e096654 to
3c27b68
Compare
3c27b68 to
8ed150d
Compare
lib/manifold.ex
Outdated
| def send(nil, _message), do: :ok | ||
|
|
||
| def current_partitioner() do | ||
| partitioner_for(self()) |
There was a problem hiding this comment.
Do we think this can ever have a situation where during a handoff stuff starts being sent to another partitioner out of order?
There was a problem hiding this comment.
no, there is no linearization requirement for handoffs across processes 👍
There was a problem hiding this comment.
Are you sure?
If a manifold is backed up on a server.. then there is a handoff on another service which changes the current pid.. you might send things to a new partitioner on the backed up server, causing messages to arrive out of order?
|
@vishnevskiy see f192229 for what we discussed in DM. |
this is just a PoC to the sharding strategy. The idea is to register 4 partitioners with a preset of names, and then send to the correct partitioner in
Manifold.sendbased off the pid of what is sending.TODO: