Expression Filter for Access Logging - `response.grpc_status` filter not working as expected #48503

mprokopjc · 2023-12-21T08:51:43Z

mprokopjc
Dec 21, 2023

👋 , I am trying to configure Istio (v1.19.3) access logging to log only non-OK HTTP & gRPC requests. I tried following configuration:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: istio-telemetry
  namespace: istio-system
spec:
  accessLogging:
  - providers:
    - name: envoy
    filter:
      expression: "response.code >= 400 || response.grpc_status > 0 || xds.cluster_name == 'BlackHoleCluster'"

but it doesn't work as expected. It logs:

all HTTP requests (including 200 OK responses) - this is unexpected (I want only 4xx and 5xx for plain HTTP requests)
gRPC requests with non-OK (>0) grpc_status - this is desired

Any suggestion?

Side note

Previously, I was using expression: "response.code >= 400 || xds.cluster_name == 'BlackHoleCluster'" and it worked as intended. It was logging just 4xx and 5xx requests. However, it didn't log any gRPC requests. When I added response.grpc_status > 0 condition later, it started to log also non-OK gRPC requests (as expected) but filtering of "plain" HTTP requests broke.

zirain · 2023-12-21T09:12:35Z

zirain
Dec 21, 2023
Collaborator

give a try with latest envoy, looks like grpc_status is 2 when http response code is 200.

accessLogFormat: "[%START_TIME%] %CEL(response.grpc_status)% \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

[2023-12-21T09:07:13.121Z] 2 "GET /get HTTP/1.1" 200 - via_upstream "-" 0 600 1 1 "-" "curl/8.5.0" "371a0dc2-c607-967f-a7b2-010d0a46f551" "httpbin:8000" "10.244.0.23:8080" outbound|8000||httpbin.default.svc.cluster.local 10.244.0.24:47768 10.96.235.15:8000 10.244.0.24:47234 - default

5 replies

zirain Dec 21, 2023
Collaborator

this's more like an envoy issue, can you kindly raise this up in envoy's repo?

mprokopjc Dec 21, 2023
Author

Thank you for a very quick response! I will raise this up in envoy's repo. In meantime, any idea how to workaround it with the version I already have (v1.19.3)? Is there any other attribute that identifies gRPC that I can use in my filter?

Following expression (response.grpc_status > 0 && request.useragent.contains("grpc")) will probably work but I don't like it so much.

zirain Dec 21, 2023
Collaborator

from https://github.com/envoyproxy/envoy/blob/0fae6970ddaf93f024908ba304bbd2b34e997a51/source/common/grpc/status.cc#L6

maybe response.grpc_status>2 is fine?

mprokopjc Dec 21, 2023
Author

Are you sure that httpToGrpcStatus comes into play in our case? I don't like this function at all 😬

It maps all 3xx and 4xx into 2 but it also maps 500 (Internal Server Error) into 2. Which means that response.grpc_status>2 filter is going to exclude also Internal Server Error(s). That is definitely not desired in my case.

zirain Dec 21, 2023
Collaborator

but it's from https://github.com/grpc/grpc/blob/master/doc/http-grpc-status-mapping.md, I don't have a better idea.

mprokopjc · 2023-12-21T12:19:50Z

mprokopjc
Dec 21, 2023
Author

Just for ref: here is my report to envoy repo envoyproxy/envoy#31483 (feel free to add anything)

0 replies

waity · 2024-02-06T17:46:59Z

waity
Feb 6, 2024

@mprokopjc request_protocol as reported by istio checks the 'content-type' header of the requests. And the GRPC spec describes:

If Content-Type does not begin with "application/grpc", gRPC servers SHOULD respond with HTTP status of 415 (Unsupported Media Type). This will prevent other HTTP/2 clients from interpreting a gRPC error response, which uses status 200 (OK), as successful.
(source)

So although it isn't pretty, has(request.headers) && 'content-type' in request.headers && request.headers['content-type'].startsWith("application/grpc") && response.grpc_status > 0 may be correct

0 replies

BrunoChauvet · 2025-10-13T18:28:29Z

BrunoChauvet
Oct 13, 2025

This is the EnvoyFilter we have provisioned to log gRPC errors based on above suggestions

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: access-logs-errors
  namespace: istio-system
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: ANY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: MERGE
        value:
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
            access_log:
              - name: envoy.access_loggers.file
                typed_config:
                  "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog"
                  path: /dev/stdout
                  typed_json_format:
                    method: '%REQ(:METHOD)%'
                    path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
                    # etc...
                    grpc_status: "%GRPC_STATUS%"
                filter:
                  or_filter:
                    filters:
                    # Only log requests with response codes >= 400 or == 0.
                    - status_code_filter:
                        comparison:
                          op: GE
                          value:
                            default_value: "400"
                            runtime_key: "400"
                    - status_code_filter:
                        comparison:
                          op: EQ
                          value:
                            default_value: "0"
                            runtime_key: "0"
                    # gRPC error have an HTTP response of 200, but have a non-zero gRPC status
                    - extension_filter:
                        name: envoy.access_loggers.extension_filters.cel
                        typed_config:
                          "@type": type.googleapis.com/envoy.extensions.access_loggers.filters.cel.v3.ExpressionFilter
                          expression: >-
                            has(request.headers) && 'content-type' in request.headers && request.headers['content-type'].startsWith("application/grpc") && response.grpc_status > 0

0 replies

Expression Filter for Access Logging - response.grpc_status filter not working as expected #48503

Uh oh!

Uh oh!

mprokopjc Dec 21, 2023

Side note

Replies: 4 comments · 5 replies

Uh oh!

zirain Dec 21, 2023 Collaborator

Uh oh!

zirain Dec 21, 2023 Collaborator

Uh oh!

mprokopjc Dec 21, 2023 Author

Uh oh!

zirain Dec 21, 2023 Collaborator

Uh oh!

mprokopjc Dec 21, 2023 Author

Uh oh!

zirain Dec 21, 2023 Collaborator

Uh oh!

mprokopjc Dec 21, 2023 Author

Uh oh!

waity Feb 6, 2024

Uh oh!

BrunoChauvet Oct 13, 2025

Expression Filter for Access Logging - `response.grpc_status` filter not working as expected #48503

mprokopjc
Dec 21, 2023

Replies: 4 comments 5 replies

zirain
Dec 21, 2023
Collaborator

zirain Dec 21, 2023
Collaborator

mprokopjc Dec 21, 2023
Author

zirain Dec 21, 2023
Collaborator

mprokopjc Dec 21, 2023
Author

zirain Dec 21, 2023
Collaborator

mprokopjc
Dec 21, 2023
Author

waity
Feb 6, 2024

BrunoChauvet
Oct 13, 2025