Use latest bijection, algebird, scalding, storehaus, chill, and tormenta, remove scala 2.9 support, merge #571 and #553 #565

ianoc · 2014-12-09T17:29:23Z

No description provided.

egonina · 2014-12-09T18:52:04Z

project/Build.scala

"Unsupported scala version error" here?

why not also pass the scalaVersion in the exception?

egonina · 2014-12-09T18:54:55Z

LGTM besides couple comments

…metrics

franklinhu · 2015-02-03T19:16:18Z

FWIW, this test failure looks like a binary incompatibility between bijection versions, and will require a new storehaus artifact.
I'm happy to help out since we'd like to get running on algebird 0.9.0 here at Stripe.

ianoc · 2015-02-03T19:21:26Z

@franklinhu Thanks for the offer, @isnotinvain is publishing storehaus at the moment I believe. We want to get running the latest Algebird here at twitter too! We are blocked by this also

isnotinvain · 2015-02-03T23:07:01Z

yes, in the process of publishing storehaus, have already published everything that comes before storehaus in the dependency graph (including scalding 0.13.0)

johnynek · 2015-02-03T23:17:12Z

.travis.yml

we can't do 2.11 until finagle get's their stuff out?

isnotinvain · 2015-02-05T02:41:44Z

Got some compile errors to resolve which I'll start wading through, but if anyone knows anything about these let me know, thanks!

[error] /Users/alexlevenson/workspace/summingbird/summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BuildSummer.scala:61: not enough arguments for constructor NullSummer: (tuplesIn: com.twitter.algebird.util.summer.Incrementor, tuplesOut: com.twitter.algebird.util.summer.Incrementor)(implicit semigroup: com.twitter.algebird.Semigroup[V])com.twitter.algebird.util.summer.NullSummer[K,V].
[error] Unspecified value parameters tuplesIn, tuplesOut.
[error]           new com.twitter.algebird.util.summer.NullSummer[K, V]
[error]           ^
[error] /Users/alexlevenson/workspace/summingbird/summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BuildSummer.scala:77: not enough arguments for constructor SyncSummingQueue: (bufferSize: com.twitter.algebird.util.summer.BufferSize, flushFrequency: com.twitter.algebird.util.summer.FlushFrequency, softMemoryFlush: com.twitter.algebird.util.summer.MemoryFlushPercent, memoryIncr: com.twitter.algebird.util.summer.Incrementor, timeoutIncr: com.twitter.algebird.util.summer.Incrementor, sizeIncr: com.twitter.algebird.util.summer.Incrementor, putCalls: com.twitter.algebird.util.summer.Incrementor, tuplesIn: com.twitter.algebird.util.summer.Incrementor, tuplesOut: com.twitter.algebird.util.summer.Incrementor)(implicit semigroup: com.twitter.algebird.Semigroup[V])com.twitter.algebird.util.summer.SyncSummingQueue[K,V].
[error] Unspecified value parameters memoryIncr, timeoutIncr, sizeIncr, ...
[error]             new SyncSummingQueue[K, V](
[error]             ^
[error] /Users/alexlevenson/workspace/summingbird/summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BuildSummer.scala:95: not enough arguments for constructor AsyncListSum: (bufferSize: com.twitter.algebird.util.summer.BufferSize, flushFrequency: com.twitter.algebird.util.summer.FlushFrequency, softMemoryFlush: com.twitter.algebird.util.summer.MemoryFlushPercent, memoryIncr: com.twitter.algebird.util.summer.Incrementor, timeoutIncr: com.twitter.algebird.util.summer.Incrementor, insertOp: com.twitter.algebird.util.summer.Incrementor, insertFails: com.twitter.algebird.util.summer.Incrementor, sizeIncr: com.twitter.algebird.util.summer.Incrementor, tuplesIn: com.twitter.algebird.util.summer.Incrementor, tuplesOut: com.twitter.algebird.util.summer.Incrementor, workPool: com.twitter.util.FuturePool, compact: com.twitter.algebird.util.summer.Compact, compatSize: com.twitter.algebird.util.summer.CompactionSize)(implicit sg: com.twitter.algebird.Semigroup[V])com.twitter.algebird.util.summer.AsyncListSum[K,V].
[error] Unspecified value parameters timeoutIncr, insertOp, insertFails, ...
[error]             val summer = new AsyncListSum[K, V](BufferSize(cacheSize.lowerBound),
[error]                          ^
[warn] /Users/alexlevenson/workspace/summingbird/summingbird-storm/src/main/scala/com/twitter/summingbird/storm/FlatMapBoltProvider.scala:50: abstract type pattern T is unchecked since it is eliminated by erasure
[warn]       case (ts: Timestamp, data: T) =>
[warn]                                  ^
[error] /Users/alexlevenson/workspace/summingbird/summingbird-storm/src/main/scala/com/twitter/summingbird/storm/StormPlatform.scala:193: value registerMetrics is not a member of com.twitter.tormenta.spout.Spout[(com.twitter.summingbird.batch.Timestamp, Any)]
[error]     val stormSpout = tormentaSpout.registerMetrics(metrics.toSpoutMetrics).getSpout
[error]                                    ^
[warn] one warning found
[error] four errors found

…veScala2.9.3 Conflicts: project/Build.scala summingbird-batch/src/main/scala/com/twitter/summingbird/batch/TimeStamp.scala

isnotinvain · 2015-02-05T02:53:50Z

Merging @MansurAshraf's #571 fixed most the issues, looking at the last one now

…2.9.3

isnotinvain · 2015-02-05T02:58:13Z

Merging @egonina's #553 fixes the last one (now I get it...)

isnotinvain · 2015-02-05T03:11:42Z

Test failures (locally, travis still seems busted):

[info] + Online Dag.Dag Nodes must be unique: OK, passed 100 tests.
[info] + Online Dag.Must have at least one producer in each MemoryNode: OK, passed 100 tests.
[info] + Online Dag.If a Node contains a Summer, all other producers must be NOP's: OK, passed 100 tests.
[info] + Online Dag.The first producer in a online node cannot be a NamedProducer: OK, passed 100 tests.
[info] + Online Dag.0 or more merge producers at the start of every online bolts, followed by 1+ non-merge producers and no other merge producers following those.: OK, passed 100 tests.
[info] + Online Dag.No producer is repeated: OK, passed 100 tests.
[info] + Online Dag.All producers are in a Node: OK, passed 100 tests.
[info] + Online Dag.Only sources can have no incoming dependencies: OK, passed 100 tests.
[info] + Online Dag.Sources must have no incoming dependencies, and they must have dependants: OK, passed 100 tests.
[info] + Online Dag.Prior to a summer the Node should be a FlatMap Node: OK, passed 100 tests.
[info] + Online Dag.There should be no flatmap producers in the source node: OK, passed 100 tests.
[info] + Online Dag.Nodes in the DAG should have unique names: OK, passed 100 tests.
[info] + Online Dag.Running through the optimizer should only reduce the number of nodes: OK, passed 100 tests.
[info] + Online Dag.The number of non-named nodes should remain constant running with StripNamedNode: OK, passed 100 tests.
[info] ! VersionBatchLaws.version -> BatchID -> version: Falsified after 9 passed tests.
[info] > ARG_0: -9223372036854775808
[info] + VersionBatchLaws.BatchID -> version -> BatchID: OK, passed 100 tests.
[info] + VersionBatchLaws.version is an upperbound on time: OK, passed 100 tests.

and

info] Total for specification ScaldingLaws
[info] Finished in 143 ms
[info] 13 examples, 0 failure, 0 error
[info] 
[error] Failed: Total 23, Failed 1, Errors 0, Passed 22
[error] Failed tests:
[error]     com.twitter.summingbird.scalding.VersionBatchLaws
[info] StormLaws
[info] 
[error] x StormPlatform matches Scala for single step jobs
[error]  the value is false (StormLaws.scala:132)
[info] + FlatMap to nothing
[error] x OptionMap and FlatMap
[error]  the value is false (StormLaws.scala:162)
[info] + OptionMap to nothing and FlatMap
[error] x StormPlatform matches Scala for large expansion single step jobs
[error]  the value is false (StormLaws.scala:194)
[error] x StormPlatform matches Scala for flatmap keys jobs
[error]  the value is false (StormLaws.scala:209)
[error] x StormPlatform matches Scala for left join jobs
[error]  the value is false (StormLaws.scala:223)
[error] x StormPlatform matches Scala for left join with flatMapValues jobs
[error]  the value is false (StormLaws.scala:238)
[info] + StormPlatform matches Scala for repeated tuple leftJoin jobs
[error] x StormPlatform matches Scala for optionMap only jobs
[error]  the value is false (StormLaws.scala:272)
[error] x StormPlatform matches Scala for MapOnly/NoSummer
[error]  List() is not equal to List(-2072127598, -2063865730, -1599548058, -1251232760, -940977214, -572841932, -2, -2, -2, 0, 0, 0, 0, 0, 0, 0, 0, 336822222, 1611910644) (StormLaws.scala:287)
[error] Expected: List(-2072127598, -2063865730, -1599548058, -1251232760, -940977214, -572841932, -2, -2, -2, 0, 0, 0, 0, 0, 0, 0, 0, 336822222, 1611910644)
[error] Actual:   List()
[info] 
[error] x StormPlatform with multiple summers
[error]  the value is false (StormLaws.scala:310)
[error] x StormPlatform should be efficent in real world job
[error]  the value is false (StormLaws.scala:353)
[info]  
[info] Total for specification StormLaws
[info] Finished in 1 minute 44 seconds, 88 ms
[info] 13 examples, 10 failures, 0 error
[info] 
[error] Failed: Total 23, Failed 10, Errors 0, Passed 13
[error] Failed tests:
[error]     com.twitter.summingbird.storm.StormLaws

MansurAshraf · 2015-02-07T01:40:35Z

Also getting

java.lang.RuntimeException: It is only valid to create counter objects during job submission
    at scala.sys.package$.error(package.scala:27)
    at com.twitter.summingbird.storm.StormStatProvider$$anonfun$counterIncrementor$1$$anonfun$apply$1.apply(StormStatProvider.scala:54)
    at com.twitter.summingbird.storm.StormStatProvider$$anonfun$counterIncrementor$1$$anonfun$apply$1.apply(StormStatProvider.scala:54)
    at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
    at scala.collection.AbstractMap.getOrElse(Map.scala:58)
    at com.twitter.summingbird.storm.StormStatProvider$$anonfun$counterIncrementor$1.apply(StormStatProvider.scala:53)
    at com.twitter.summingbird.storm.StormStatProvider$$anonfun$counterIncrementor$1.apply(StormStatProvider.scala:51)
    at scala.Option.map(Option.scala:145)
    at com.twitter.summingbird.storm.StormStatProvider$.counterIncrementor(StormStatProvider.scala:51)
    at com.twitter.summingbird.SummingbirdRuntimeStats$$anonfun$getPlatformCounterIncrementor$1$$anonfun$apply$2.apply(Stats.scala:73)
    at com.twitter.summingbird.SummingbirdRuntimeStats$$anonfun$getPlatformCounterIncrementor$1$$anonfun$apply$2.apply(Stats.scala:72)
    at scala.Option.flatMap(Option.scala:170)
    at com.twitter.summingbird.SummingbirdRuntimeStats$$anonfun$getPlatformCounterIncrementor$1.apply(Stats.scala:72)
    at com.twitter.summingbird.SummingbirdRuntimeStats$$anonfun$getPlatformCounterIncrementor$1.apply(Stats.scala:71)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
    at com.twitter.summingbird.SummingbirdRuntimeStats$.getPlatformCounterIncrementor(Stats.scala:71)
    at com.twitter.summingbird.Counter.incrCounter$lzycompute(Counter.scala:34)
    at com.twitter.summingbird.Counter.incrCounter(Counter.scala:33)
    at com.twitter.summingbird.Counter.incrBy(Counter.scala:36)
    at com.twitter.summingbird.Counter.incr(Counter.scala:38)
    at com.twitter.algebird.util.summer.NullSummer$$anonfun$1.apply(NullSummer.scala:32)
    at com.twitter.algebird.util.summer.NullSummer$$anonfun$1.apply(NullSummer.scala:31)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at com.twitter.algebird.GenericMapMonoid.sumOption(MapAlgebra.scala:77)
    at com.twitter.algebird.Semigroup$.sumOption(Semigroup.scala:76)
    at com.twitter.algebird.util.summer.NullSummer.addAll(NullSummer.scala:31)
    at com.twitter.summingbird.online.executor.Summer.apply(Summer.scala:121)
    at com.twitter.summingbird.online.executor.Summer.apply(Summer.scala:56)
    at com.twitter.summingbird.online.executor.AsyncBase.execute(AsyncBase.scala:45)
    at com.twitter.summingbird.online.executor.AsyncBase.execute(AsyncBase.scala:26)
    at com.twitter.summingbird.storm.BaseBolt.execute(BaseBolt.scala:138)
    at backtype.storm.daemon.executor$fn__3557$tuple_action_fn__3559.invoke(executor.clj:610)
    at backtype.storm.daemon.executor$mk_task_receiver$fn__3480.invoke(executor.clj:382)
    at backtype.storm.disruptor$clojure_handler$reify__3017.onEvent(disruptor.clj:43)
    at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
    at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
    at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
    at backtype.storm.daemon.executor$fn__3557$fn__3569$fn__3616.invoke(executor.clj:715)
    at backtype.storm.util$async_loop$fn__436.invoke(util.clj:377)

@ianoc I suspect that is because we are lazily creating lazy val sCache = summerBuilder.getSummer[SummerK, SummerV](implicitly[Semigroup[(Seq[S], Value)]]) and the counters are getting initialized inside legacyBuilder. Instead of creating counters in BuildSummer should we move them one level up? Please note this is only failing for NullSummer during test runs

…cala2.9.3

…ault job id.

MansurAshraf · 2015-02-10T21:05:06Z

@isnotinvain all tests that were failing before are passing, build is failing because of excessive output/ length of the build.

isnotinvain · 2015-02-10T21:34:46Z

Actually looks like the job didn't output anything for > 10 minutes. Could be that a thread kept the jvm alive after the tests finished. It sort of looks like the tests finished though (assuming the storm tests are the last ones)

isnotinvain · 2015-02-10T21:35:05Z

@ianoc @MansurAshraf do you think this gives us enough confidence to merge?

ianoc · 2015-02-10T21:37:36Z

If you can do an sbt test locally and it passes i think lets keep going

On Tue, Feb 10, 2015 at 1:35 PM, Alex Levenson [email protected]
wrote:

@ianoc https://github.com/ianoc @MansurAshraf
https://github.com/MansurAshraf do you think this gives us enough
confidence to merge?

—
Reply to this email directly or view it on GitHub
#565 (comment).

johnynek · 2015-02-10T22:11:31Z

should we use the matrix approach to separate the storm tests from the rest of the tests?

isnotinvain · 2015-02-11T04:55:52Z

@johnynek yes, probably, though we've decided to do that separately. I'm going to merge this and do a release now.

Use latest bijection, algebird, scalding, storehaus, chill, and tormenta, remove scala 2.9 support, merge #571 and #553

Katya Gonina and others added 3 commits October 16, 2014 15:28

pass a function to the metric registration on the spout

dda66a7

openHook to do all metric registration in one function

b2a7f82

Re-order build file, kill 2.9.3 entry

03366bb

egonina reviewed Dec 9, 2014
View reviewed changes

MansurAshraf and others added 11 commits December 12, 2014 16:35

added counters

5d35e98

added jobid

a29ed67

made counter creation eager

5166063

refactored counters

ce645b3

removed redundant put counter

a2767c9

bumped algebird version

ece1481

Merge remote-tracking branch 'remotes/upstream/develop' into feature/…

436fa69

…metrics

fix build on 2.10

8494e96

Merge branch 'develop' into removeScala2.9.3

f71024c

Fixup Build

e0078f2

changed bunch of case classes to AnyVal

96f540e

johnynek reviewed Feb 3, 2015
View reviewed changes

.travis.yml

Copy link

Collaborator

johnynek Feb 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can't do 2.11 until finagle get's their stuff out?

Use latest bijection, algebird, scalding, storehaus, chill, and tormenta

73e9e51

isnotinvain changed the title ~~Re-order build file, kill 2.9.3 entry~~ Re-order build file, kill 2.9.3 entry, Use latest bijection, algebird, scalding, storehaus, chill, and tormenta Feb 5, 2015

isnotinvain mentioned this pull request Feb 5, 2015

Add counters to default summers #571

Merged

Merge remote-tracking branch 'MansurAshraf/feature/metrics' into remo…

9d1e1b7

…veScala2.9.3 Conflicts: project/Build.scala summingbird-batch/src/main/scala/com/twitter/summingbird/batch/TimeStamp.scala

Merge remote-tracking branch 'origin/spout_counters' into removeScala…

8f1cef6

…2.9.3

isnotinvain mentioned this pull request Feb 5, 2015

Registering summingbird counters with tormenta spouts #553

Merged

isnotinvain changed the title ~~Re-order build file, kill 2.9.3 entry, Use latest bijection, algebird, scalding, storehaus, chill, and tormenta~~ Use latest bijection, algebird, scalding, storehaus, chill, and tormenta, remove scala 2.9 support, merge #571 and #553 Feb 5, 2015

ianoc added 2 commits February 6, 2015 15:09

Fix storm tests

a61785a

Make val private

f3ad616

MansurAshraf and others added 5 commits February 9, 2015 10:46

added job id generation

688cb10

Merge remote-tracking branch 'upstream/removeScala2.9.3' into removeS…

9193ba2

…cala2.9.3

Fixes tests

a461951

Fixes logging in storm tests, change what we mean by putting in a def…

6265208

…ault job id.

removed println

6ad058e

isnotinvain added a commit that referenced this pull request Feb 11, 2015

Merge pull request #565 from twitter/removeScala2.9.3

21d193d

Use latest bijection, algebird, scalding, storehaus, chill, and tormenta, remove scala 2.9 support, merge #571 and #553

isnotinvain merged commit 21d193d into develop Feb 11, 2015

Use latest bijection, algebird, scalding, storehaus, chill, and tormenta, remove scala 2.9 support, merge #571 and #553 #565

Use latest bijection, algebird, scalding, storehaus, chill, and tormenta, remove scala 2.9 support, merge #571 and #553 #565

Uh oh!

Conversation

ianoc commented Dec 9, 2014

Uh oh!

egonina Dec 9, 2014

Choose a reason for hiding this comment

Uh oh!

johnynek Feb 3, 2015

Choose a reason for hiding this comment

Uh oh!

egonina commented Dec 9, 2014

Uh oh!

franklinhu commented Feb 3, 2015

Uh oh!

ianoc commented Feb 3, 2015

Uh oh!

isnotinvain commented Feb 3, 2015

Uh oh!

johnynek Feb 3, 2015

Choose a reason for hiding this comment

Uh oh!

isnotinvain commented Feb 5, 2015

Uh oh!

isnotinvain commented Feb 5, 2015

Uh oh!

isnotinvain commented Feb 5, 2015

Uh oh!

isnotinvain commented Feb 5, 2015

Uh oh!

MansurAshraf commented Feb 7, 2015

Uh oh!

MansurAshraf commented Feb 10, 2015

Uh oh!

isnotinvain commented Feb 10, 2015

Uh oh!

isnotinvain commented Feb 10, 2015

Uh oh!

ianoc commented Feb 10, 2015

Uh oh!

johnynek commented Feb 10, 2015

Uh oh!

isnotinvain commented Feb 11, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants