Support leftJoin against a store in offline platform #557

egonina · 2014-11-13T21:52:17Z

Adds support for performing a leftJoin against a store in the offline platform. Currently supports two cases:

sumByKey(store) is not dependent on the leftJoin
sumByKey(store) has a limited dependency on the leftJoin - only legal nodes between the join and store are [map | filter | flatMap]Values, i.e. we cannot have any key fanout

Based off of @johnynek's PR: #551

… memory tests

… platforms

…PNamedProducer in DagOptimizer

…tore check

johnynek · 2014-11-14T06:17:39Z

summingbird-core/src/test/scala/com/twitter/summingbird/TestGraphs.scala

don't you want to use your sum on line 44 here?

ah, yes, thanks :)

…rgedProducer from options for nodes between join and store

egonina · 2014-11-18T22:31:27Z

This is ready for review. Thanks!

johnynek · 2014-11-20T01:14:51Z

summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/LookupJoin.scala

why is this here?

leftover from debugging.. thanks

johnynek · 2014-11-20T01:31:05Z

summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/Service.scala

isn't this always Any => TraversableOnce[Any]?

Yes, we can make the type more constrained

egonina · 2014-11-22T20:05:44Z

Thanks @johnynek for the comments! This is ready for another pass.
@ianoc could you take a look as well, please?

ianoc · 2014-11-24T19:06:03Z

summingbird-core/src/main/scala/com/twitter/summingbird/memory/Memory.scala

needs an issue? or ?

johnynek · 2014-11-25T20:03:40Z

summingbird-core/src/main/scala/com/twitter/summingbird/Producer.scala

I thought we ditched using KeyedProducer in the input position. Can we change this to Producer[P, (K, V)] and get everything to compile.

johnynek · 2014-11-25T21:34:46Z

Added a handful of minor nits, then I'm ready to merge.

I do agree with Ian that the TODOs should have issue's added and we should reference those issue numbers in the comment.

egonina · 2014-11-27T01:11:23Z

Addressed @johnynek's comments except need to address unit batcher & scalding tests. Also I made the modifications for that TODO, no issue needed, but will file issues next time.

Support leftJoin against a store in offline platform

…r#557)

Katya Gonina added 12 commits November 6, 2014 16:01

working off of oscar's branch, case 1 (store indep of join) working +…

072e254

… memory tests

change back to InternalService.leftDoesNotDependOnStore

107fba9

ValueFlatMapToFlatMap dag optimizer rule to Memory & ConcurrentMemory…

e1ed916

… platforms

work in progress for case 2

835e77e

getLoopInputs

870cb9a

getLoopInputs with multiple flatMapValues functions

f02c1d7

mergedProducer for join-store dep case

cf58430

case2 working with multiple merges

a8a521b

cleaning up ScaldingLaws tests

35acad3

proper handling of new ValueFlatMappedProducer node in PlannerSpec; T…

fc950c5

…PNamedProducer in DagOptimizer

cleaning up + must have at least one flatMapValues between join and s…

e995d64

…tore check

fixing type compilation error in scala 2.10

8054fb3

johnynek reviewed Nov 14, 2014
View reviewed changes

Katya Gonina added 5 commits November 17, 2014 16:15

Removing semigroup from BatchedStore (get from Summer now); Remove Me…

ba26fc7

…rgedProducer from options for nodes between join and store

Adding test for fanout after join (dep on store)

b36e027

Spark platform to handle ValueFlatMappedProducer

5ef65ad

invalid loop sys.error; cleaning up

ee436f8

minor tweaks

fd690c9

johnynek reviewed Nov 20, 2014
View reviewed changes

Katya Gonina added 2 commits November 21, 2014 17:34

Addressing most of Oscar's comments; todo getLoopInputs

899481e

getLoopInputs Option[(Any) => TraversableOnce[Any])

a71ee9e

egonina mentioned this pull request Nov 24, 2014

Support MergedProducer in join against dependent store for offline platform #559

Open

ianoc reviewed Nov 24, 2014
View reviewed changes

summingbird-core/src/main/scala/com/twitter/summingbird/memory/Memory.scala Outdated

Copy link

Collaborator

ianoc Nov 24, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs an issue? or ?

egonina mentioned this pull request Nov 24, 2014

Handle a diamond case for jobs that join against a dependent store (offline) #560

Open

JMemoryService class; addressing Ian's comments

d41da7b

johnynek reviewed Nov 25, 2014
View reviewed changes

addressing oscar's comments

a391d86

ianoc added a commit that referenced this pull request Nov 27, 2014

Merge pull request #557 from twitter/kgonina/leftjoin_against_store

1b63069

Support leftJoin against a store in offline platform

ianoc merged commit 1b63069 into develop Nov 27, 2014

ianoc deleted the kgonina/leftjoin_against_store branch November 27, 2014 01:31

johnynek mentioned this pull request Nov 27, 2014

First step to support join against an internal store in scalding #551

Closed

snoble pushed a commit to snoble/summingbird that referenced this pull request Sep 8, 2017

remove seed variable in BloomFilter and rename k to hashIndex (twitte…

534bdfb

…r#557)

Uh oh!

Support leftJoin against a store in offline platform #557

Support leftJoin against a store in offline platform #557

Uh oh!

Conversation

egonina commented Nov 13, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

egonina commented Nov 18, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

egonina commented Nov 22, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnynek commented Nov 25, 2014

Uh oh!

egonina commented Nov 27, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants