Description

This PR addresses the lack of observability in the Feast Go Feature Server by implementing comprehensive metrics for both HTTP and gRPC endpoints, and integrating them with the existing OpenTelemetry tracing.

Changes

• Metrics Implementation: Added Prometheus metrics using gotoprom (a type-safe wrapper for the official Prometheus client) to ensure standardized bucket configuration and label safety.
  • Added internal/feast/metrics/metrics.go to define histograms and counters.
  • Exposed metrics via the /metrics endpoint.
• HTTP Observability:
  • Implemented a middleware in internal/feast/server/http_server.go to capture request duration and status codes.
  • Note on Design: The middleware logic is encapsulated within http_server.go (instead of main.go) to maintain high cohesion within the server struct and ensure metrics are applied correctly to business endpoints while excluding health checks.
• gRPC Observability:
  • Added a Unary Interceptor in main.go to capture gRPC specific metrics (latency, status codes, methods).
  • Configured a sidecar HTTP server on port 9090 to expose Prometheus metrics when running in gRPC mode.
• Tracing:
  • Integrated OpenTelemetry spans within the HTTP handlers to ensure trace propagation.

How Has This Been Tested?

• Manual Testing: Ran the server in both HTTP and gRPC modes.
  • Verified /metrics endpoint returns valid Prometheus data.
  • Verified /get-online-features increments the counters and updates histograms.
  • Verified panic recovery middleware works as expected without crashing the server.
• Unit Tests: Added/Verified tests in metrics_test.go.

Checklist

• My code follows the code style of this project.
• I have added tests that prove my fix is effective or that my feature works.
• I have updated the dependencies (go.mod / go.sum).

Why gotoprom

It reduces boilerplate code for histograms and ensures type-safety for labels, preventing runtime panics due to mismatched label cardinality. It wraps the official prometheus client, so it's fully compatible