Add CustomMetricsOps #7469

dj707chen · 2024-06-17T21:28:17Z

Add CustomMetricsOps and update client and server Metrics middleware to allow adding custom labels to metrics.

Background
In a situation where a service interacts with multiple providers, a few of them do not have a lot of traffic, so when one of these light providers is down, the alert of the service is not triggered due to the small percentage counted in the failing traffic; So we want to create alerts specific to providers, and I’m searching for ways to identify the metrics entries related to a provider.

I can think of two options currently available:

Use provider specific prefix when create MetricsOps instance, but the metrics of this client will be excluded from a generic Http4S metrics dashboard;
Add provider specific prefix to classifier when create org.http4s.client.middleware.Metrics.

The second option is better than the first option, but not ideal.

What it does
This PR

Add CustomMetricsOps (extends existing MetricsOps) to allow adding custom labels to metrics records;
Add withCustomLabels method to both client and server Metrics middleware to pass custom label values when they call the CustomMetricsOps with custom label values.

When CustomMetricsOps with some custom labels is created, the Metrics middleware calling it must supply the same number of label values, so the type parameter SL (sized list) is added to enforce this constraint.

We could use shapeless's Sized type to express this constraint, but shapeless is not on the dependency list and it's unlikely it will be added. So we temporarily created SizedSeq to mimic shapeless's Sized type.

http4s-prometheus-metrics will be updated to adapt this feature, a draft pull request on http4s-prometheus-metrics Allow custom labels allows creating instance of CustomMetricsOps with custom label. epimetheus-http4s could also be updated.

Code example
First call http4s-prometheus-metrics's Prometheus to create an instance of CustomMetricsOps, notice the prefix payments:org_http4s_client, and one single custom label payment_provider.
Then create a Metrics client middleware, passing value Paylly to custom label.

for {
    # Create a CustomMetricsOps using the new method to be added to http4-promethues-metrics
    # with one custom label "payment_provider"
    customMetricsOps: CustomMetricsOps[F, SizedSeq1[String]] <- Stream.resource(
        Prometheus
          .default(cr)
          .withPrefix("payments:org_http4s_client")
          .buildCustomMetricsOps(CustomLabels(SizedSeq1("payment_provider"), SizedSeq1("default value1")))
      )
    # Create client middleware calling the customMetricsOps, passing custom label value "Paylly"
    client <- BlazeClientBuilder[F]
        .resource
        .map { client =>
            Metrics.withCustomLabels(
                customMetricsOps,
                SizedSeq1("Paylly")   # SizedSeq2("Paylly", "Other") would cause compilation error here
                (req: Request[F]) => req.uri.host.map(_.value))(client)
        }
}

Output examples
I created SNAPSHOT version of http4s and ran test in local, the above code generated below metrics entries; Notice that the entries with prefix payments:org_http4s_client_ have custom label payment_provider with value Paylly.

# Normal metrics entries:
org_http4s_client_response_duration_seconds_count{classifier="localhost",method="get",phase="body",} 3.0
org_http4s_client_response_duration_seconds_sum{classifier="localhost",method="get",phase="body",} 0.511480755
org_http4s_client_response_duration_seconds_bucket{classifier="localhost",method="get",phase="headers",le="0.005",} 0.0

# Custom metrics entries with custom label payment_provider:
payments:org_http4s_client_response_duration_seconds_bucket{classifier="test.com",method="post",phase="headers",payment_provider="Paylly",le="0.005",} 0.0
payments:org_http4s_client_response_duration_seconds_bucket{classifier="test.com",method="post",phase="headers",payment_provider="Paylly",le="+Inf",} 1.0
payments:org_http4s_client_response_duration_seconds_count{classifier="test.com",method="post",phase="headers",payment_provider="Paylly",} 1.0
payments:org_http4s_client_response_duration_seconds_sum{classifier="test.com",method="post",phase="headers",payment_provider="Paylly",} 0.526104997
payments:org_http4s_client_response_duration_seconds_count{classifier="test.com",method="post",phase="body",payment_provider="Paylly",} 1.0
payments:org_http4s_client_response_duration_seconds_sum{classifier="test.com",method="post",phase="body",payment_provider="Paylly",} 0.571559018
payments:org_http4s_client_request_count_total{classifier="test.com",method="post",status="2xx",payment_provider="Paylly",} 1.0
payments:org_http4s_client_request_count_created{classifier="test.com",method="post",status="2xx",payment_provider="Paylly",} 1.717544234533E9
payments:org_http4s_client_response_duration_seconds_created{classifier="test.com",method="post",phase="headers",payment_provider="Paylly",} 1.717544234488E9
payments:org_http4s_client_response_duration_seconds_created{classifier="test.com",method="post",phase="body",payment_provider="Paylly",} 1.717544234533E9

rossabaker · 2024-06-17T22:19:08Z

I spoke with @dj707chen about this. I think the basic tradeoff here is:

This is reinventing things that epimetheus core already does, specifically having properly sized custom labels.
But we can't just do this in epimetheus-http4s without duplicating all the HTTP metric functionality that's already http4s's Metrics middlewares.

dj707chen · 2024-06-17T23:32:22Z

But we can't just do this in epimetheus-http4s without duplicating all the HTTP metric functionality that's already http4s's Metrics middlewares.

The http4s Metrics middlewares currently interact with MetricsOps only and there is no means to pass custom labels; the new CustomMetricsOps extends MetricsOps, adding methods with custom label values, below is increaseActiveRequests as example:

trait CustomMetricsOps[F[_], SL <: SizedSeq[String]] extends MetricsOps[F] {
  def definingCustomLabels: CustomLabels[SL]

  def increaseActiveRequests(classifier: Option[String], customLabelValues: SL): F[Unit]
  override def increaseActiveRequests(classifier: Option[String]): F[Unit] =
    increaseActiveRequests(classifier, definingCustomLabels.values)

dj707chen · 2024-06-17T23:43:05Z

This is reinventing things that epimetheus core already does, specifically having properly sized custom labels.

epimetheus core's interface is defined using shapless's Sized type. I guess we don't want to add shapeless as a dependency for http4s core, I had to create the repetitive SizedSeq.

iRevive · 2024-07-23T09:18:07Z

When it comes to the labels, there are two scenarios:

Constant labels: payment_provider=Paylly or payment_provider=Wise
Dynamic labels: decide the set of labels based on the request, e.g. if (request.host.contains("paylly")) "payment_provider=Paylly" else "payment_provider=Wise"

In some sense, constant labels are dynamic labels where _ => "payment_provider=Paylly".

Also, it seems that MetricsOps design was heavily influenced by integration with Prometheus. However, OpenTelemetry is gaining more traction each year, becoming a standard in the industry.

However, each metric backend treats labels differently. For example, in Prometheus, labels are key-value string entries.
In OpenTelemetry, labels (attributes) are typed.

While working with Prometheus, you must ensure that the configured instrument labels Counter.builder().labelNames("a", "b", "c") match the provided one: counter.labelValues("a value", "b value", "c value").inc(). That's why the SizedSeqN is mandatory to ensure correctness at the compile time.

And that's not the case for OpenTelemetry because you can use different sets of attributes (labels) with the same instrument, unlike Prometheus:

The current concept of the classifier is outdated because it's limited to exactly one label.
Currently, the classifier can be configured only through the Metrics:

object Metrics {
  def apply[F[_]](
      ops: MetricsOps[F],
      classifierF: Request[F] => Option[String] = { (_: Request[F]) => None },
  )(client: Client[F])(...): Client[F] = 
}

Which makes it pretty limiting.

I believe the backend should provide a way to build a classifier. For example:

trait MetricsOps[F[_]] {
  type Classifier
  def classifier(request: RequestPrelude): F[Option[Classifier]]
  def increaseActiveRequests(request: RequestPrelude, classifier: Option[Classifier]): F[Unit]
  ...
}

class OtelMetrics[F[_]] extends MetricsOps[F] {
  type Classifier = Attributes
}

class PrometheusMetrics[F[_]] extends MetricsOps[F] {
  type Classifier = Labels
}

Or even different classifiers per metric (which might be an overkill):

trait MetricsOps[F[_]] {
  type Classifier
  def classifier(metric: Metric, request: RequestPrelude): F[Option[Classifier]]
  def increaseActiveRequests(request: RequestPrelude, classifier: Option[Classifier]): F[Unit]
  ...
}

sealed trait Metric
object Metrics {
  case object ActiveRequests extends Metric
  case object TotalTime extends Metric
  case object RequestSize extends Metric
  case object ResponseSize extends Metric
}

dj707chen · 2024-07-24T18:28:58Z

@iRevive Thanks for the insightful comment, I think the changes proposed in your comment could be good candidate for next major version.

hamnis

I think this is a reasonable change given it has to be binary compatible.

Leaving it up to other maintainers to merge.

core/shared/src/main/scala/org/http4s/metrics/CustomLabels.scala

rossabaker

I share the concern that this is biasing the type toward Prometheus, but I have thought about this off and on and not come up with a better way.

As we work on instrumenting backends directly with otel4s (e.g., #7476), I think that the MetricsOps will become less important anyway.

I'm good with this, with a few small nitpicks below.

rossabaker · 2024-08-09T22:12:19Z

core/shared/src/main/scala/org/http4s/metrics/CustomLabels.scala

+    }
+}
+
+final case class EmptyCustomLabels() extends CustomLabels[SizedSeq0[String]] {


It seems like this could be a case object.

Tried to convert, but had difficulty, see my comment below on SizedSeq0.

Changed it to abstract and the constructor of EmptyCustomLabels private.

core/shared/src/main/scala/org/http4s/util/SizedSeq.scala

rossabaker · 2024-08-09T22:17:57Z

core/shared/src/main/scala/org/http4s/util/SizedSeq.scala

+}
+
+// format: off
+final case class SizedSeq0[+A]() extends SizedSeqBase[A] {


I think this could be a case object.

Tried to convert it to a case object, but encountered difficulty, I may tried a wrong way?

case object SizedSeq0 { def toSeq[A]: Seq[A] = Seq.empty[A] }

and EmptyCustomLabels will look like this:

case object EmptyCustomLabels { val labels = SizedSeq0 val values = SizedSeq0 }

Then CustomMetricsOps.fromMetricsOps will looks like:

def fromMetricsOps[F[_]](ops: MetricsOps[F]): CustomMetricsOps[F, SizedSeq0.type] = {

and it got compilation error:

Type SizedSeq0.type does not conform to upper bound SizedSeq[String] of type parameter SL

Root cause is that when we define SizedSeq0 as object, it can't extend SizedSeq since object can't define a type parameter which SizedSeq requires.

So I'm leaving it as case class for now.

Ah, I didn't think of that. Could do something like this:

final case class SizedSeq0[+A] private () extends SizedSeq[A] { val toSeq: Seq[A] = Seq.empty[A] } final object SizedSeq0 { private[this] val instance = SizedSeq0[Nothing]() def apply[A](): SizedSeq0[A] = instance }

It would be the only SizedSeq that can't be created with new, but it would prevent us from creating multiple instances when a singleton does just fine. I'm not sure if it's worth it.

Made the change.

Made it also abstract to fix the following error:

[error] /home/runner/work/http4s/http4s/core/shared/src/main/scala/org/http4s/util/SizedSeq.scala:24:1: error: [Http4sGeneralLinters.nonValidatingCopyConstructor] Case classes with private constructors should be abstract to prevent exposing a non-validating copy constructor [error] final case class SizedSeq0[+A] private () extends SizedSeq[A] { [error] ^ [error] (coreJS / scalafixAll) scalafix.sbt.ScalafixFailed: LinterError

core/shared/src/main/scala/org/http4s/util/SizedSeq.scala

…q11 - SizedSeq22

…LinterError

…tomMetricsOps

rossabaker

🎉

dj707chen added 6 commits June 4, 2024 11:58

Add SizedSeq to be used by the coming CustomMetricsOps

8b054da

Add CustomMetricsOps

e923abd

Update client and server Metrics middleware for CustomMetricsOps

d580a74

Run prePR

9a5bef4

Run scalafixAll

fd9c6ea

Run scalafixAll - fix SizedSeq.scala

820c1dc

mergify bot added series/0.23 PRs targeting 0.23.x module:core module:server module:client labels Jun 17, 2024

mergify bot mentioned this pull request Jun 17, 2024

Add CustomMetricsOps #7451

Closed

Merge branch 'series/0.23' into addCustomMetricsOps

d565d25

dj707chen marked this pull request as ready for review June 17, 2024 21:36

dj707chen mentioned this pull request Jun 17, 2024

Update to use CustomMetricsOps http4s/http4s-prometheus-metrics#168

Merged

rossabaker added this to the 0.23.28 milestone Jun 17, 2024

Run scalafixAll - fix Metrics.scala

89b2d0e

dj707chen added 6 commits June 18, 2024 10:27

Merge branch 'series/0.23' into addCustomMetricsOps

46e62ef

Merge branch 'series/0.23' into addCustomMetricsOps

a9018c8

Merge branch 'series/0.23' into addCustomMetricsOps

5ccd26b

Merge branch 'series/0.23' into addCustomMetricsOps

fac9f1f

Merge branch 'series/0.23' into addCustomMetricsOps

37904a4

Merge branch 'series/0.23' into addCustomMetricsOps

36b60f1

hamnis approved these changes Jul 30, 2024

View reviewed changes

Merge branch 'series/0.23' into addCustomMetricsOps

d32bba9

danicheg reviewed Aug 4, 2024

View reviewed changes

core/shared/src/main/scala/org/http4s/metrics/CustomLabels.scala Show resolved Hide resolved

dj707chen requested a review from danicheg August 5, 2024 15:03

Merge branch 'series/0.23' into addCustomMetricsOps

38d80e5

rossabaker reviewed Aug 9, 2024

View reviewed changes

Addressing PR comments: Remove intermediate SizedSeqBase, add SizedSe…

dfb8e29

…q11 - SizedSeq22

dj707chen requested a review from rossabaker August 10, 2024 03:20

dj707chen added 6 commits August 14, 2024 08:52

Make constructor of SizedSeq0 private

a020b8b

Merge branch 'series/0.23' into addCustomMetricsOps

5f82e99

Make constructor of SizedSeq0 private - make it also abstract to fix …

5220bbf

…LinterError

Merge remote-tracking branch 'origin/addCustomMetricsOps' into addCus…

0b860cb

…tomMetricsOps

Make constructor of EmptyCustomLabels private

d11fdfc

Make constructor of EmptyCustomLabels private - clean up

3d3520a

rossabaker approved these changes Aug 16, 2024

View reviewed changes

rossabaker merged commit 1ec9d1d into http4s:series/0.23 Aug 16, 2024

Add CustomMetricsOps #7469

Add CustomMetricsOps #7469

Uh oh!

Conversation

dj707chen commented Jun 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rossabaker commented Jun 17, 2024

Uh oh!

dj707chen commented Jun 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dj707chen commented Jun 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iRevive commented Jul 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dj707chen commented Jul 24, 2024

Uh oh!

hamnis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rossabaker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rossabaker left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dj707chen commented Jun 17, 2024 •

edited

Loading

dj707chen commented Jun 17, 2024 •

edited

Loading

dj707chen commented Jun 17, 2024 •

edited

Loading

iRevive commented Jul 23, 2024 •

edited

Loading