Experimenting with new ways to provide data to reporters #37

eric · 2013-04-17T07:14:03Z

This is a basic attempt at figuring out what it would look like to give some
intermediate mechanisms to make reporters simpler.

This is a basic attempt at figuring out what it would look like to give some intermediate mechanisms to make reporters simpler.

eric · 2013-04-17T07:14:43Z

CC: @bernd, @lmarburger

eric · 2013-04-17T07:15:32Z

One thing that I've realized is that the CounterDerivative and FlatteningRegistryEnumerator classes could practically be combined into a single class with a conditional in the initializer.

eric · 2013-04-17T07:17:12Z

lib/metriks/reporter/counter_derivative.rb

If we only report like this, it means we won't actually report the marks that happened before the reporter was run for the first time.

Because we know these things are in-process, I think it would actually make more sense to initialize @last with a default of 0 for the hash values:

@last = Hash.new do |h,k| h[k] = 0 end

eric · 2013-04-17T07:19:09Z

This is a completely different attempt to solve the same problems that were raised by #21 and #35.

eric · 2013-04-17T07:33:41Z

lib/metriks/reporter/flattening_registry_enumerator.rb

I'm wondering if it ever makes sense to actually send count... in the cases that I can think of, I don't believe it does.

eric · 2013-04-17T07:34:57Z

I've just made a couple other changes to possibly make FlatteningRegistryEnumerator do everything.

bernd · 2013-04-17T07:54:44Z

lib/metriks/reporter/flattening_registry_enumerator.rb

This should be :counter_derivative.

bernd · 2013-04-17T08:08:42Z

lib/metriks/reporter/flattening_registry_enumerator.rb

The #median method only exists on the snapshot object. This doesn't work. (and #median is already called on the snapshot below)

Same for Timer and Histogram.

Good catch.

eric · 2013-07-02T18:31:06Z

I'm convinced now that CounterDerivative should be removed and that FlatteningRegistryEnumerator should have an option to specify if it reports derivative counts or rates.

eric · 2013-07-04T23:59:07Z

Okay. I've made some improvements that feel good to me.

I have some outstanding issues:

1. Filtering

It seems clear we should implement some sort of filtering — it seems that it should be done in a standard way for all reporters.

2. Metric name changes

I've made some changes to some of the metrics that were returned.

For instance, before, a gauge called "special.gauge", would be reporting a value as "special.gauge.value". With this change, it now just calls it "special.gauge".

The rational was that there was no real benefit of having something appended to the metric name when there is only one value that would be returned.

Is this change going to mess people up and bother them? Should I make this optional?

3. Change from sending rates

I've changed the default that is reported to report the number of pieces of work that have been completed in the reporting interval instead of the exponentially weighted rate per second.

It does this by recording the last count and submitting the difference instead of sending the one_minute_rate.

Does it make sense to change the default here?

myronmarston · 2013-07-15T18:03:48Z

I pointed our Gemfile to this branch to try this out over the weekend, and deployed it friday evening. Unfortunately, we got no data in librato at all over the weekend. I changed back to the latest metriks gem release (0.9.9.5) and now data is showing up again. So...there's something in here that completely stopped librato from receiving data from us. Not sure what yet. Digging into it now. @eric, if you have any ideas or if this is a known issue, please let me know!

myronmarston · 2013-07-15T18:06:12Z

I talked with the librato guys in their campfire room, and they found this in their logs:

Response: [{"errors":{"params":{"name":["[\"qless:job-times:Packrat::BackendAdapters::Silo::EventFeedProcessor::Job.999th_percentile\", 0.694730768, Metriks::Timer] is not a valid metric name"]}}}]

Looks like the data is being submitted with name as the JSON dump of an array rather than the actual name. I'll look into this a bit more.

eric · 2013-07-15T18:08:09Z

I'll look into this shortly. Sorry about that.

eric · 2013-07-15T21:17:33Z

Okay. I've fixed the problem and added a test to make sure I don't make the mistake again.

Thanks!

myronmarston · 2013-07-15T21:25:45Z

👍

Thanks, I'll give it another go.

Prevent 400 error from librato when submitting an empty body

lmarburger · 2013-11-01T14:57:15Z

I noticed Librato supports count, sum, min, max, and sum_squares on a gauge. Do we want to take advantage of those as part of this PR?

eric · 2013-11-01T20:08:15Z

For which metric types?

lmarburger · 2013-11-01T21:25:34Z

I suppose we'd have to switch to reporting count, min, max, etc. for a given duration instead of for the lifetime of the process.

roidrage · 2014-03-28T15:33:08Z

Hey, can I help test this out?

I remember we also talked about adding this as a second Librato reporter to at least keep the old behaviour around.

What do you think?

eric · 2014-03-28T16:51:31Z

There are a couple things that are missing from the librato reporter that I would like to get fixed:

Send extra metadata the first time we send a metric that defines that:
- it is a server-side aggregated metric
- the period
- the minimum

Then the only thing to figure out is a migration strategy for people (if that is needed). The metric names will be changing for people as it is now, which isn't great, but there are a couple issues if they don't change:

Librato doesn't provide a way to easily update things like the "server side aggregation" bit on existing metrics when you send new datapoints
The units of the metrics will be different (from rates per second to total work done in a period)

To test this, one simple approach would be to point your Gemfile at this branch and run it.

roidrage · 2014-03-28T16:57:29Z

I can look into adding these things, I'm keen to try it out.

eric · 2014-03-28T16:58:28Z

My idea was to hold onto a hash of metrics we've sent before, and if the hash doesn't contain a key, send additional :attributes containing those things.

That way it won't send larger payloads more often than it may need to.

…ntation Conflicts: lib/metriks/reporter/librato_metrics.rb

Experimenting with providing data to reporters

a3be900

This is a basic attempt at figuring out what it would look like to give some intermediate mechanisms to make reporters simpler.

eric reviewed Apr 17, 2013
View reviewed changes

Look at combining the counter tracking

f6c6de4

eric reviewed Apr 17, 2013
View reviewed changes

bernd reviewed Apr 17, 2013
View reviewed changes

lib/metriks/reporter/flattening_registry_enumerator.rb Outdated

Copy link

Contributor

bernd Apr 17, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be :counter_derivative.

Spelling

e729482

bernd reviewed Apr 17, 2013
View reviewed changes

josephruscio mentioned this pull request Jun 27, 2013

Custom percentile reporting #41

Closed

eric added 2 commits July 4, 2013 16:41

Implement changes based on discussions

d824753

Add require

a4afd3d

Ensure everything gets submitted properly

0b6e337

Andrew and others added 2 commits October 4, 2013 18:08

Prevent 400 error from librato by submitting an empty body

9994eba

Merge pull request #43 from GreyKn/empty_response_librato

6e50126

Prevent 400 error from librato when submitting an empty body

Merge remote-tracking branch 'origin/master' into enumerator-experime…

8980d03

…ntation Conflicts: lib/metriks/reporter/librato_metrics.rb

Experimenting with new ways to provide data to reporters #37

Are you sure you want to change the base?

Experimenting with new ways to provide data to reporters #37

Uh oh!

Conversation

eric commented Apr 17, 2013

Uh oh!

eric commented Apr 17, 2013

Uh oh!

eric commented Apr 17, 2013

Uh oh!

eric Apr 17, 2013

Choose a reason for hiding this comment

Uh oh!

bernd Apr 17, 2013

Choose a reason for hiding this comment

Uh oh!

eric commented Apr 17, 2013

Uh oh!

eric Apr 17, 2013

Choose a reason for hiding this comment

Uh oh!

eric commented Apr 17, 2013

Uh oh!

bernd Apr 17, 2013

Choose a reason for hiding this comment

Uh oh!

bernd Apr 17, 2013

Choose a reason for hiding this comment

Uh oh!

eric Apr 18, 2013

Choose a reason for hiding this comment

Uh oh!

eric commented Jul 2, 2013

Uh oh!

eric commented Jul 4, 2013

1. Filtering

2. Metric name changes

3. Change from sending rates

Uh oh!

myronmarston commented Jul 15, 2013

Uh oh!

myronmarston commented Jul 15, 2013

Uh oh!

eric commented Jul 15, 2013

Uh oh!

eric commented Jul 15, 2013

Uh oh!

myronmarston commented Jul 15, 2013

Uh oh!

lmarburger commented Nov 1, 2013

Uh oh!

eric commented Nov 1, 2013

Uh oh!

lmarburger commented Nov 1, 2013

Uh oh!

roidrage commented Mar 28, 2014

Uh oh!

eric commented Mar 28, 2014

Uh oh!

roidrage commented Mar 28, 2014

Uh oh!

eric commented Mar 28, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants