Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Jan 20, 2022. It is now read-only.

Platform-independent stats storm scalding #503

Merged
merged 31 commits into from
May 22, 2014
Merged

Platform-independent stats storm scalding #503

merged 31 commits into from
May 22, 2014

Conversation

@egonina
Copy link
Contributor

@egonina egonina commented May 12, 2014

First stab at the platform-independent stats in Summingbird. Not ready for merge, open for comments. Still working on unit testing.

isnotinvain and others added 13 commits March 28, 2014 15:12
@isnotinvain
Merge branch 'release/0.4.2'
a78a2b7
Initial platform-independent stats, working on storm backend
e80db66
Initial refactor to add scalding stats. Testing that storm still works
60d2997
First stab at scalding and storm stats
62775e9
Debugging
8987f23
Merge branch 'master' of github.com:twitter/summingbird into pi_stats…
40a3e0b 
…_storm_scalding
Adding lazy object instantiator for scalding platform
ee3edd8
Forcing instantiation of ScaldingPlatformProvider object
e3f6e8b
moving jobID option fron namedOption to transformConfig
c012a12
adding group to statistics
b17a1a5
use SynchronizedSet for platformMetricProviders for thread-safety
b8c90d9
Merge branch 'develop' of github.com:twitter/summingbird into pi_stat…
02c9636 
…s_storm_scalding
Try{} around getting statistics for Bolt - if no stats passed, return…
6276c8d 
… empty list
jcoveney
jcoveney reviewed May 13, 2014
View reviewed changes
summingbird-core/src/main/scala/com/twitter/summingbird/Stats.scala Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do we stand on author tags? They are in some places, aren't in others. seems like git blame would be good enough to me but I dunno. @johnynek @ianoc @sritchie ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we use them at all in the SB OSS do we?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't think so.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed author

ianoc
ianoc reviewed May 21, 2014
View reviewed changes
summingbird-scalding/src/main/scala/com/twitter/summingbird/scalding/ScaldingStatProvider.scala Outdated
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

private?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't make this private, ScaldingStatProvider refers to it

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make it package private then?

ianoc
ianoc reviewed May 21, 2014
View reviewed changes
summingbird-storm/src/main/scala/com/twitter/summingbird/storm/BaseBolt.scala Outdated
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wildcard import for what?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to be explicit imports

@ianoc
Copy link
Collaborator

ianoc commented May 21, 2014

LGTM when green. Build failure looked probably spurious so i've restarted it.

@egonina
Copy link
Contributor Author

egonina commented May 21, 2014

All green, tested on a cluster both storm and scalding, things work fine. Ready for merge.

ianoc
ianoc reviewed May 22, 2014
View reviewed changes
summingbird-core/src/main/scala/com/twitter/summingbird/Stats.scala Outdated
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Package private?

@ianoc
Copy link
Collaborator

ianoc commented May 22, 2014

Awesome work, thanks! This is huge!

ianoc added a commit that referenced this pull request May 22, 2014
@ianoc
Merge pull request #503 from twitter/pi_stats_storm_scalding
11cc4c4 
Platform-independent stats storm scalding
@ianoc ianoc merged commit 11cc4c4 into develop May 22, 2014
@ianoc ianoc deleted the pi_stats_storm_scalding branch May 22, 2014 02:37
johnynek
johnynek reviewed May 24, 2014
View reviewed changes
summingbird-core/src/main/scala/com/twitter/summingbird/Counter.scala
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be really nice if we could count different things, or name groups differently on each platform.

Something like:

def job[P <: Platform[P]](source: P#Source[T], sink: P#Sink[V], counters: P#Counters): TailProducer[Unit] =
  source.map { t =>
    counters("mapStep").inc
    fn(t)
  }.write(sink)

And then I count configure each of the counters by name on each platform (and maybe not do anything at all on some).

Might be something to think of down the line (platform specific counters).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@egonina @ianoc @johnynek @ryanlecompte @jcoveney @sriramkrishnan @isnotinvain