Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fixed request counting with subrequests in case of error. #515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 20, 2025

Conversation

pluknet
Copy link
Contributor

@pluknet pluknet commented Feb 12, 2025

Previously, when creating a subrequest, request count increment could be missed in case of error after it became active. This resulted in "http request count is zero" alerts and socket leaks.

Further, if posting the next subrequest resulted in an error, then as part of the main request termination, request pool could be destroyed from a subrequest cleanup handler due to the wrong request reference count, followed by a segmentation fault.

Found by bad memory allocator simulation.

@pluknet pluknet requested a review from arut February 12, 2025 13:59
@arut
Copy link
Contributor

arut commented Feb 13, 2025

Could you please elaborate more on how exactly to reproduce this issue? From the branch name it follows that the primary way to do it is by using the ssi module. Just triggering the error in the code and running ssi.t is not enough though. There should be a setting where the malformed subrequest is taken from c->data and processed. After finalizing the subrequest, the counter is decremented and the main request is processed. Finalizing the main request will trigger the error.

Normally, a subrequest error finalizes the entire main request (see addition for example). In this case c->data is usually not accessed, unless mr->blocked is set (see ngx_http_terminate_request) but that's a rare case. Changing the counter behavior would damage this case since the subrequest is not registered anywhere but in c->data. SSI module however is an exception where a failed subrequest is ignored and may remain active and potentially lead to the consequences described.

Looks like a simple way to access a request from c->data is limit_rate. Adding a simple limit_rate to the test did not trigger the issue though, probably more finetuning is needed.

@pluknet
Copy link
Contributor Author

pluknet commented Feb 19, 2025

Indeed, both cases are explained by proxied subrequests initiated by SSI "virtual" command,
with the below sample config, where ssi.html contains included (sub)request(s) matched by "location /proxy-":

        location /ssi.html {
            ssi on;
        }

        location /proxy- {
            proxy_pass http://u/;
        }

In the basic scenario, with a single failing subrequest, request count zero alerts are papered over by r->count reset to 1 in ngx_http_terminate_handler().
Here, after ngx_http_subrequest() error, the processing will end up in send timeout that will kick in ngx_http_request_handler() for the given active subrequest, where it will initiate request proxying. After its normal processing and finalization, a parent request will be awaken and, due to send timeout, it will get into ngx_http_terminate_request() with request count zero. Here it is posted with ngx_http_terminate_handler() that resets r->count to 1. No socket leaks, no alerts.

With two included requests in ssi.html the behaviour is different, such as in:

<!--#include virtual="/proxy-index.html?1" -->
<!--#include virtual="/proxy-index.html?2" -->

Here, after failing 1st subrequest, the 2nd is run posted. Then the 1st (active) subrequest will be normally run on upstream write event. Eventually, after normal processing and finalizing both subrequests, r->count is zero.

In addition to the above, consider the 1st failing subrequest finished its processing first (or it served static content), and a parent request was awaken, where it then posts 2nd subrequest from the postpone filter. Here, ngx_http_post_request() additional error results in main request termination with terminate request count:1. This calls in turn all cleanup handlers including registered from subrequests. Such one is ngx_http_upstream_cleanup() for the 2nd subrequest, which may end up in ngx_http_finalize_request(NGX_DONE), and due to the wrong request count, in ngx_http_close_request() for the main request, which ultimately will destroy request pool. Sanitizer catches this U-A-F on the next cln->next, otherwise it may or may not segfault depending on how far the freed request was damaged.

@pluknet
Copy link
Contributor Author

pluknet commented Feb 19, 2025

SSI module however is an exception where a failed subrequest is ignored and may remain active and potentially lead to the consequences described.

Changing SSI module behaviour to return NGX_ERROR looks like a separate change, which might have pitfails.
For now, I suggest to fix ngx_http_subrequest() initialization order to eliminate inconsistency (see updated patch).

@arut
Copy link
Contributor

arut commented Feb 20, 2025

The patch looks ok, as discussed before. However, I think the commit log needs a clarification about the fact that the issue only manifests itself with the ssi module, which ignores ngx_http_subrequest() errors.

As mentioned above, fixing the ssi module to treat such errors as fatal could be an alternative solution. However, ngx_http_subrequest() leaving the requst in an inconsistent state is a problem as well.

Previously, request might be left in inconsistent state in case of error,
which manifested in "http request count is zero" alerts when used by SSI
filter.

The fix is to reshuffle initialization order to postpone committing state
changes until after any potentially failing parts.

Found by bad memory allocator simulation.
@pluknet
Copy link
Contributor Author

pluknet commented Feb 20, 2025

The patch looks ok, as discussed before. However, I think the commit log needs a clarification ...

Agree, I don't like commit log either.
Please see updated.

Copy link
Contributor

@arut arut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok.

@pluknet pluknet merged commit d25139d into nginx:master Feb 20, 2025
1 check passed
@pluknet pluknet deleted the ssi branch February 20, 2025 20:04
@Maryna-f5 Maryna-f5 added this to the nginx-1.27.5 milestone Feb 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants