Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Tests fail because expected exceptions are intercepted by ExceptionController #33478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
derrabus opened this issue Sep 5, 2019 · 11 comments
Closed

Tests fail because expected exceptions are intercepted by ExceptionController #33478

derrabus opened this issue Sep 5, 2019 · 11 comments

Comments

@derrabus
Copy link
Member

derrabus commented Sep 5, 2019

Symfony version(s) affected: 4.4-dev

Description
#33271 introduced an error controller that renders error pages for exceptions. Before that, TwigBundle rendered error pages and if that bundle was missing, the exception bubbled up to the code that called Kernel::handle().

The new behavior might be more consistent, however it breaks tests that relied on the old behvior, like this one from SecurityBundle 4.3:

symfony/src/Symfony/Bundle/SecurityBundle/Tests/Functional/MissingUserProviderTest.php

Lines 16 to 26 in 828e5a4

public function testUserProviderIsNeeded()
{
$this->expectException('Symfony\Component\Config\Definition\Exception\InvalidConfigurationException');
$this->expectExceptionMessage('"default" firewall requires a user provider but none was defined.');
$client = $this->createClient(['test_case' => 'MissingUserProvider', 'root_config' => 'config.yml']);
$client->request('GET', '/', [], [], [
'PHP_AUTH_USER' => 'username',
'PHP_AUTH_PW' => 'pa$$word',
]);
}

The test fails if FrameworkBundle 4.4 is installed and if I downgrade FrameworkBundle to 4.3, the test is green.

PR #33271 patches the test on the 4.4 branch to parse the error page instead of catching the exception, but we cannot do that on 4.3 because then the test would not work with FrameworkBundle 4.3 anymore.

Also, this kind of looks like a BC break to me. Userland tests might run into similar issues.

ping @yceruto

How to reproduce
https://travis-ci.org/symfony/symfony/jobs/580647626
@yceruto yceruto mentioned this issue Sep 5, 2019
Merged
@derrabus
Copy link
Member Author

derrabus commented Sep 5, 2019

Thanks @yceruto, PR #33479 fixes the issue for the Symfony test suite. Would this also be the recommended way to fix userland tests that fail because of this issue? Is this documented somewhere?

@yceruto
Copy link
Member

yceruto commented Sep 5, 2019 •
edited
Loading

To me, test cases like this one ($client = $this->createClient(...) + $client->request(...)) shouldn't be tested against $this->expectException(...) that's not the real live, you should expect a real error response instead. That's why I think it is not a BC break.

It works in 4.3 because the exception handler is replaced by PHPUnit and the response instance is never instantiated.

To fix this in your project you can to add the TwigBundle or activate the HttpKernel's ExceptionListener service for your tests suite with a proper exception controller.

@yceruto
Copy link
Member

yceruto commented Sep 5, 2019

To fix this in your project you can to add the TwigBundle or activate the HttpKernel's ExceptionListener service for your tests suite with a proper exception controller.

The last, is what we have by default now as of 4.4

@yceruto
Copy link
Member

yceruto commented Sep 5, 2019

I faced this issue in #31398 (comment), but the other way around. See for example these tests in 4.3 https://github.com/symfony/symfony/blob/4.4/src/Symfony/Bundle/SecurityBundle/Tests/Functional/JsonLoginTest.php

@derrabus
Copy link
Member Author

derrabus commented Sep 5, 2019

To me, test cases like this one ($client = $this->createClient(...) + $client->request(...)) shouldn't be tested against $this->expectException(...) that's not the real live, you should expect a real error response instead. That's why I think it is not a BC break.

Well, tests aren't real live. And for functional tests like this one, a developer might want to check what exception specifically is raised. That's a bit complicated when parsing a human-readable error page. So, I can to a certain point understand why in general tests like this one have been built. For this specific test, the error page is fine though.

Anyway, I'm rather concerned about test suites breaking when people start upgrading to Symfony 4.4. A documented migration path would be helpful.

@yceruto
Copy link
Member

yceruto commented Sep 5, 2019 •
edited
Loading

If a developer wants to check which exception is specifically raised, it is likely that they will not create a functional test with a request/response approach for that purpose, but a unit-specific test.

I'm not really convinced why we would need a migration path for this kind of Functional test that in first place shouldn't happen.

@yceruto
Copy link
Member

yceruto commented Sep 5, 2019

That's a bit complicated when parsing a human-readable error page.

It doesn't seem so complicated IMHO, see https://github.com/symfony/symfony/pull/33479/files#diff-e84e5e830de34b1539d8bf85ed6042b5R27

@yceruto
Copy link
Member

yceruto commented Sep 5, 2019

And related to the title: "Tests fail because expected exceptions are intercepted by ExceptionController", I can say that "Tests pass because expected exceptions are not intercepted by ExceptionController" ;) that would be the correct behavior expected for these functional tests.

The question now is: does it deserves a patch?

@derrabus
Copy link
Member Author

derrabus commented Sep 5, 2019

And related to the title: "Tests fail because expected exceptions are intercepted by ExceptionController", I can say that "Tests pass because expected exceptions are not intercepted by ExceptionController" ;) that would be the correct behavior expected for these functional tests.

Yes, the old behavior was rather an accident than documented behavior. I really like the new behavior because Kernel::handle() with $catch = true now consistently catches exceptions as one might expect. That's a very good change and I thank you a lot for having implemented that!

But the old behavior had been there since 2.x. My concern is that this very well crafted feature might break tests and leave developers clueless about what's wrong. Yes, those tests should maybe have been written differently in the first place.

The question now is: does it deserves a patch?

No, but it deserves documentation, imho.

@yceruto
Copy link
Member

yceruto commented Sep 5, 2019 •
edited
Loading

No, but it deserves documentation, imho.

Completely agree, thanks!

I just added a comment in symfony/symfony-docs#12273 to not forget it.

nicolas-grekas added a commit that referenced this issue Sep 6, 2019
@nicolas-grekas
minor #33479 [SecurityBundle] Fix 4.3 tests forward compat (yceruto)
2feae7b 
This PR was merged into the 4.3 branch.

Discussion
----------

[SecurityBundle] Fix 4.3 tests forward compat

| Q             | A
| ------------- | ---
| Branch?       | 4.3
| Bug fix?      | no
| New feature?  | no
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets | #33478
| License       | MIT
| Doc PR        | -

Commits
-------

f092331 Fix 4.3 tests forward compat
@derrabus
Copy link
Member Author

derrabus commented Sep 6, 2019

I think we can close here. The "broken" test has been fixed and upgrade path will be documented with symfony/symfony-docs#12273.

@derrabus derrabus closed this as completed Sep 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@derrabus @yceruto