Let's talk about Exception and Log #38901

lyrixx · 2020-10-30T15:19:08Z

This issue is a bit long, and cover two topics, to please read it all before
commenting.

1️⃣ Do not log anymore 4XX exceptions

With a fresh Symfony (5.1) but it's the same for older version, all exceptions
are logged. It could be are at error level or at critical level.

I created a simple controller:

<?php

namespace App\Controller;

use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Exception\BadRequestException;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpKernel\Exception\BadRequestHttpException;
use Symfony\Component\Routing\Annotation\Route;

class HomepageController extends AbstractController
{
    /** @Route("/", name="homepage") */
    public function index(): Response
    {
        throw new BadRequestException();
    }

    /** @Route("/2", name="homepage_2") */
    public function index2(): Response
    {
        throw new BadRequestHttpException();
    }
}

And when I hit the application, I got the following log (prod env)

[2020-10-30T15:52:01.636868+01:00] request.INFO: Matched route "homepage". {"route":"homepage","route_parameters":{"_route":"homepage","_controller":"App\\Controller\\HomepageController::index"},"request_uri":"https://127.0.0.1:8000/","method":"GET"} []
[2020-10-30T15:52:01.637921+01:00] request.ERROR: Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\BadRequestHttpException: "" at /tmp/tmp/vendor/symfony/http-kernel/HttpKernel.php line 82 {"exception":"[object] (Symfony\\Component\\HttpKernel\\Exception\\BadRequestHttpException(code: 0):  at /tmp/tmp/vendor/symfony/http-kernel/HttpKernel.php:82)\n[previous exception] [object] (Symfony\\Component\\HttpFoundation\\Exception\\BadRequestException(code: 0):  at /tmp/tmp/src/Controller/HomepageController.php:16)"} []

[2020-10-30T15:52:04.005203+01:00] request.INFO: Matched route "homepage_2". {"route":"homepage_2","route_parameters":{"_route":"homepage_2","_controller":"App\\Controller\\HomepageController::index2"},"request_uri":"https://127.0.0.1:8000/2","method":"GET"} []
[2020-10-30T15:52:04.009468+01:00] request.ERROR: Uncaught PHP Exception Symfony\Component\HttpKernel\Exception\BadRequestHttpException: "" at /tmp/tmp/src/Controller/HomepageController.php line 22 {"exception":"[object] (Symfony\\Component\\HttpKernel\\Exception\\BadRequestHttpException(code: 0):  at /tmp/tmp/src/Controller/HomepageController.php:22)"} []

For recall, from the monolog
documentation

ERROR (400): Runtime errors that do not require immediate action but should typically be logged and monitored.

So there is two things here:

Symfony logged at the error level, meaning I should monitor this, and of
course, fix it.
My application decided to throw a 400. And 4XX are for user errors. This
means my code is good, the request is not. => There is nothing I can do to
fix it

This is mismatch to me.

This is even more obvious with 404's. This is why, in the past we introduced in
monolog the options excluded_404s.

And then we also add excluded_http_codes because, well, almost no one want
logs for 405, 403, 401, etc.

And there is also activation_strategy to have full control. This is what I use
in my application for ages. And I copy this code from projets to projects. Boring

So my proposable it to not log at error level HTTP exceptions < 500. We could
make it opt-in, but IMHO, this is a good default.

2️⃣ HttpException in the "model layer".

When we create controller, we have the power to manage exceptions handling, and
forge the response the use the status code we want.

But when we use third party code, like easy-admin, api-platform, etc, usually we
don't write the controller by ourself. So we can not really convert an exception
to a status code easily.

Api-platform offer us an option to convert exception instance to status code,
but I would like to go a bit further here.

First I'm not very confortable to throw *HttpException from
symfony/http-kernel
in my model layer. It does not sens when I use theses service in my commands, or
workers.

I would feel more natural to have "raw" exception, without a binding to the SAPI.
Something like:

NotFoundException
ForbidenException
InvalidInputException
etc

Symfony would, of course, convert a NotFoundException to a 404 when needed.

And now, we have a nice new error handlers (with serializer), so we could
easily throw raw exceptions, and have a nice handling. APIP could even remove
it's own handling from its codebase. Like:

And we could also have a map of custom exception to status code (and a way to
enable log or not for such exception, see 1️⃣)

WDYT ?

The text was updated successfully, but these errors were encountered:

julienfalque · 2020-10-30T15:32:59Z

My application decided to throw a 400. And 4XX are for user errors. This
means my code is good, the request is not. => There is nothing I can do to
fix it

This is generally true, but sometimes it shows that one of your page contains a broken link (404) or that a server-side validation does not match the way a form or API endpoint is implemented (400). For that reason I think it would be nice to:

not have those errors at the same place as other errors because it mostly creates noise;
be able to still log them somewhere else easily.

HttpException in the "model layer".

Shouldn't this remain part of the domain of your application? What would be the value of having this in Symfony?

dunglas · 2020-10-30T15:50:22Z

A big 👍 on my side.

@julienfalque this kind of analytics is better done at the web server level.

lyrixx · 2020-10-30T17:54:17Z

@julienfalque I agree with @dunglas here. 404 should be monitored with [nginx|apache|lb|varnsih] logs. Or your can use externals tools like https://redirection.io/

About 400, I usually monitor that from the client POV (JS)

Shouldn't this remain part of the domain of your application? What would be the value of having this in Symfony?

An automatic conversion of NotFound to 404, AccessDenied to 403, etc. Like it's now

ro0NL · 2020-10-31T10:42:20Z

what's the proposed namespace/package for NotFoundException & co?

i've always created custom exceptions (either super specific, or generic) to be used at the domain level. Then convert them to HTTP context, yes.

see also api-platform/core#3294, maybe it's related on high level.

julienfalque · 2020-10-31T10:48:22Z

@julienfalque this kind of analytics is better done at the web server level.

Agreed, but I would still keep the ability to log them via Symfony in a separate chanel, in case e.g. one cannot access logs of the webserver.

ro0NL · 2020-10-31T10:57:49Z

btw, what semantically bugs me is

use Symfony\Component\HttpFoundation\Exception\BadRequestException;
use Symfony\Component\HttpKernel\Exception\BadRequestHttpException;

just looking at it, it's confusing already :}

derrabus · 2020-10-31T13:03:37Z

My application decided to throw a 400. And 4XX are for user errors. This
means my code is good, the request is not. => There is nothing I can do to
fix it

You cannot generalize this assumption. I've worked in projects where we 100% owned the clients that interacted with the Symfony app (REST-API for an SPA or mobile app). So in that case, a 4XX in most cases means:

Someone's trying to do nasty things.
The client submitted something broken. This is a problem that needs to be fixed.
The 4XX error is a false positive and the Symfony app has to be fixed.

In the case that the frontend or backend side of my app has a problem, I'd like to get a log message including the exception(s) with stack trace(s). Logging those exceptions is reasonable here. Logging 4xx via external monitoring only means I'm losing debug information.

This is even more obvious with 404's.

If my client tries to access an endpoint that doesn't exist, I'd like to know. If my backend hides a resource that the client thinks it should have access to, I'd like to know.

So my proposable it to not log at error level HTTP exceptions < 500. We could
make it opt-in, but IMHO, this is a good default.

We probably agree that it should be easier for the app developer to do the right thing. But I don't think that we achieve this simply by swapping defaults.

For me, the main problem is that the severity of an exception is highly subjective. It's okay for the framework to have an opinion, but as an app developer I want to be able to override that opinion. Maybe the solution is to create a better way to tell the framework the severity of a given exception.

First I'm not very confortable to throw *HttpException from
symfony/http-kernel
in my model layer. It does not sens when I use theses service in my commands, or
workers.

I agree 100%. Your model layer should not have an opinion on HTTP status codes.

Symfony would, of course, convert a NotFoundException to a 404 when needed.

I don't see how throwing an exception that will be converted to a 404 is better than directly throwing a HTTP exception.

If your model layer throws a NotFoundException, it means that who ever interacted with our model layer requested data that is not there. The model layer does not know, who's made the mistake here. It could be that the data was requested on behalf of the client, but it could also mean that the developer forgot to handle a certain edge case. If you turned all uncaught NotFoundExceptions into 404s – combined with ignoring 404s during exception logging – you're hiding potential defects of the application.

jkufner · 2020-11-01T11:34:34Z

404 errors matter when you want to know about broken links. However, it is enough to know about this link once. For such a scenario it would be useful to have a hash table, where URL and HTTP code are the key. The stored values would be a counter, how many times the error happened, and first and last time of occurrence.

derrabus · 2020-11-01T12:18:02Z

@jkufner Clustering of log messages is a feature of a good log management system. I don‘t think that the app should take care of this.

miniyarov · 2020-11-06T22:11:23Z

This comes down to somewhat opinionated log management. Apart from what to log, an app should also have a solid where to log configuration. I blogged our solution for a scaled at application https://medium.com/@miniyarov/an-opinionated-symfony-monolog-configuration-for-prod-and-dev-environments-b33487fd9ea6

carsonbot · 2021-05-07T13:04:23Z

Thank you for this suggestion.
There has not been a lot of activity here for a while. Would you still like to see this feature?

lyrixx · 2021-05-07T14:58:48Z

Yes, I planned to do that soon

javiereguiluz added MonologBridge Feature RFC RFC = Request For Comments (proposals about features that you want to be discussed) labels Oct 31, 2020

carsonbot added the Stalled label May 7, 2021

carsonbot removed the Stalled label May 7, 2021

lyrixx mentioned this issue Jul 23, 2021

[HttpKernel] Add support for configuring log level, and status code by exception class #42244

Merged

fabpot closed this as completed in e026dc5 Sep 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Let's talk about Exception and Log #38901

Let's talk about Exception and Log #38901

lyrixx commented Oct 30, 2020

julienfalque commented Oct 30, 2020

dunglas commented Oct 30, 2020

lyrixx commented Oct 30, 2020

ro0NL commented Oct 31, 2020 •

edited

Loading

julienfalque commented Oct 31, 2020

ro0NL commented Oct 31, 2020

derrabus commented Oct 31, 2020 •

edited

Loading

jkufner commented Nov 1, 2020

derrabus commented Nov 1, 2020

miniyarov commented Nov 6, 2020

carsonbot commented May 7, 2021

lyrixx commented May 7, 2021

Let's talk about Exception and Log #38901

Let's talk about Exception and Log #38901

Comments

lyrixx commented Oct 30, 2020

1️⃣ Do not log anymore 4XX exceptions

2️⃣ HttpException in the "model layer".

julienfalque commented Oct 30, 2020

dunglas commented Oct 30, 2020

lyrixx commented Oct 30, 2020

ro0NL commented Oct 31, 2020 • edited Loading

julienfalque commented Oct 31, 2020

ro0NL commented Oct 31, 2020

derrabus commented Oct 31, 2020 • edited Loading

jkufner commented Nov 1, 2020

derrabus commented Nov 1, 2020

miniyarov commented Nov 6, 2020

carsonbot commented May 7, 2021

lyrixx commented May 7, 2021

ro0NL commented Oct 31, 2020 •

edited

Loading

derrabus commented Oct 31, 2020 •

edited

Loading