OpenSSL ECH integration #840

sftcd · 2025-08-12T21:33:46Z

Proposed changes

This PR adds Encrypted Client Hello (ECH) functionality to NGINX, when using OpenSSL for TLS.

This addresses #266

Notes:

ECH is not yet part of an OpenSSL release. We'd hope ECH will be part of OpenSSL 4.0 in April 2026. However, we have been working with OpenSSL maintainers on the so-called "ECH feature branch" and that branch (subject to the same OpenSSL maintainer approval process as the OpenSSL master branch) now includes sufficient ECH code for web servers like NGINX. So there's plenty of time for this PR to be discussed, but starting now may be timely.
This PR includes documentation in a markdown document in the repo's top directory, which is certainly the wrong place, but may be useful short-term. That describes how to do the build, configuration and logging changes, and the code changes for ECH. (So should be a good place for reviewers to start.)
While OpenSSL releases do not yet include ECH support, some other TLS libraries do, in particular boringssl. If useful, we could extend this PR to also support boringssl, or that could be a follow-up. (It'd be good if the server configuration were the same regardless of the TLS library.)
ECH support using these OpenSSL ECH APIs was included in the ligthttpd web server (in January 2025) so some code and patterns are common with that. We also plan to submit similar PRs to apache2 and haproxy, and ideally all would share some commonality.
All that said, we're not fixated at all on things being done this way, and would be happy to make whatever changes are desired for NGINX and there are some notes on potential changes in the documentation.

Lastly, for open-ness, our work on this has been funded by the Open Technology Fund (OTF) in the DEfO project.

yaroslavros · 2025-09-06T19:49:13Z

Excellent work. ECH would be very useful in nginx.

My biggest concern with proposed approach is that this integration makes it virtually impossible to use nginx ECH with other TLS libraries such as BoringSSL or AWS-LC. They have no notion of ECHCONFIG PEM format. I would strongly recommend to have a simple configuration similar to ssl_ech *public_name* *config_id* *[key=file]* [noretry] proposed in yaroslavros@99fbe14 and manage these parameters inside nginx configuration instead of a dedicated directory with base-64 packaged file. I believe this would be more inline with the rest of nginx configuration approach.

sftcd · 2025-09-06T20:13:34Z

Thanks for taking a look!

Excellent work. ECH would be very useful in nginx.

My biggest concern with proposed approach is that this integration makes it virtually impossible to use nginx ECH with other TLS libraries such as BoringSSL or AWS-LC.

That's not correct. Lighttpd supports both OpenSSL and boring with this format. That's s simple enough shim to handle the file format at the application layer. See here for the code. I'd be happy to do similarly for nginx should boring support be considered a requirement for this PR. (Which'd not be unreasonable.)

They have no notion of ECHCONFIG PEM format. I would strongly recommend to have a simple configuration similar to ssl_ech *public_name* *config_id* *[key=file]* [noretry] proposed in yaroslavros@99fbe14 and manage these parameters inside nginx configuration instead of a dedicated directory with base-64 packaged file. I believe this would be more inline with the rest of nginx configuration approach.

I really don't think we want an ECH config_id inside an nginx config file - cloudflare rotate their ECH keys hourly (as do I for my servers) so it's quite likely other deployments will do similarly.

Lastly, the apache2 maintainers seemed fine with this aspect in the comments they've made on the almost equivalent PR to this one.

ECH-build.md

src/event/ngx_event_openssl.c

yaroslavros · 2025-09-06T20:21:30Z

Thanks for taking a look!

Excellent work. ECH would be very useful in nginx.
My biggest concern with proposed approach is that this integration makes it virtually impossible to use nginx ECH with other TLS libraries such as BoringSSL or AWS-LC.

That's not correct. Lighttpd supports both OpenSSL and boring with this format. That's s simple enough shim to handle the file format at the application layer. See here for the code. I'd be happy to do similarly for nginx should boring support be considered a requirement for this PR. (Which'd not be unreasonable.)

They have no notion of ECHCONFIG PEM format. I would strongly recommend to have a simple configuration similar to ssl_ech *public_name* *config_id* *[key=file]* [noretry] proposed in yaroslavros@99fbe14 and manage these parameters inside nginx configuration instead of a dedicated directory with base-64 packaged file. I believe this would be more inline with the rest of nginx configuration approach.

I really don't think we want an ECH config_id inside an nginx config file - cloudflare rotate their ECH keys hourly (as do I for my servers) so it's quite likely other deployments will do similarly.

Lastly, the apache2 maintainers seemed fine with this aspect in the comments they've made on the almost equivalent PR to this one.

Thanks for the correction! Indeed, not hardcoding config_id into configuration is very beneficial and having a history of making a compatibility shim for other libraries completely mitigates my key concern :-)

sftcd · 2025-09-06T21:10:33Z

push above includes the obvious things from @yaroslavros comments today

sftcd · 2025-09-12T16:23:19Z

As an FYI, the corresponding apache httpd functionality has been committed there: apache/httpd@0c9cd09

arut · 2025-09-25T10:34:10Z

I'd be happy to do similarly for nginx should boring support be considered a requirement for this PR. (Which'd not be unreasonable.)

Indeed we do need BoringSSL support.

Also, reading a PEM file in nginx is only a half of the problem. The other half is how to create those files with BoringSSL?

arut · 2025-09-25T10:35:27Z

Regarding the code, @sftcd could you please follow the nginx code style, see https://github.com/nginx/nginx/blob/master/CONTRIBUTING.md for details.

arut · 2025-09-25T10:39:03Z

src/event/ngx_event_openssl.c

+    case SSL_ECH_STATUS_NOT_TRIED:
+        snprintf(buf,PATH_MAX, "not attempted");
+        break;
+    case SSL_ECH_STATUS_FAILED:
+        snprintf(buf, PATH_MAX, "tried but failed");
+        break;
+    case SSL_ECH_STATUS_BAD_NAME:
+        snprintf(buf, PATH_MAX, "worked but bad name");
+        break;
+    case SSL_ECH_STATUS_SUCCESS:
+        snprintf(buf, PATH_MAX, "success");
+        break;
+    case SSL_ECH_STATUS_GREASE:
+        snprintf(buf, PATH_MAX, "GREASEd ECH");
+        break;
+    case SSL_ECH_STATUS_BACKEND:
+        snprintf(buf, PATH_MAX, "Backend/inner ECH");
+        break;
+    default:
+        snprintf(buf, PATH_MAX, "error getting ECH status");
+        break;


I'm not sure PATH_MAX is related to these values. Moreover, I suggest short upper-case status strings (NOT_TRIED, FAILED etc), see ngx_ssl_get_client_verify() for example.

ack, will do

The values do not fully match, eg SSL_ECH_STATUS_BAD_NAME -> WORKED_BAD_NAME, SSL_ECH_STATUS_GREASE -> GREASED.

Also, could you please explain what the SSL_ECH_STATUS_BACKEND status means. I looked into the code and it looks like the flag only makes sense in split mode.

sftcd · 2025-09-25T13:36:24Z

@arut: happy to make changes along the above lines (maybe tomorrow before I get time to start that), but I've a question first: over the last week or so, the freenginx project has added ECH support taking a slightly different approach to what I did with this PR, the main difference is that they chose to configure file names for the ECH PEM files rather than a directory name (from which those ECH PEM files are read) as is done here. I don't know how or if nginx and freenginx try to maintain config file or other compatibility, but probably good if you tell me that you prefer following the file name or directory name approach before we go much further.

Some details can be seen here

sftcd · 2025-09-26T22:21:30Z

I'd be happy to do similarly for nginx should boring support be considered a requirement for this PR. (Which'd not be unreasonable.)

Indeed we do need BoringSSL support.

Totally reasonable. If it's ok I'd suggest trying to agree how to use the OpenSSL ECH APIs first, then add a shim to make the same thing work with boring (or AWS-LC). That'd be simpler for me at least:-) Anyway, I'd be happy to follow such a plan.

Also, reading a PEM file in nginx is only a half of the problem. The other half is how to create those files with BoringSSL?

That's easy enough. I knocked up a small bash script that shows how.

sftcd · 2025-09-26T23:06:16Z

Regarding the code, @sftcd could you please follow the nginx code style, see https://github.com/nginx/nginx/blob/master/CONTRIBUTING.md for details.

Had tried to, and now have tried again (seeing that I'd missed some). Might take a few goes to be quite right, but happy to get there. Next push will include that.

sftcd · 2025-09-26T23:18:03Z

push above tries to address @arut comments above, modulo when to do what about boring etc. (and of course my mistakes in trying to do the right thing:-)

arut

Also don't forget to do all this for the Stream module as well.

arut · 2025-10-09T11:34:28Z

src/event/ngx_event_openssl.h

+/* check defines from <openssl/ssl.h> for ECH support */
+#if !defined(SSL_OP_ECH_GREASE)
+#define OPENSSL_NO_ECH
+#endif
+#ifndef OPENSSL_NO_ECH
+#include <openssl/ech.h>
+#endif


Why do we need this block? As I see, openssl/ech.h is included automatically by openssl/ssl.h.

Regarding checking OPENSSL_NO_ECH in nginx code for conditional compilation. Since ECH may not be available at all, relying solely on this macro is not a good idea. Modifying this macro is not a good idea either. We already have one feature that's compiled conditionally and may not be available at all, which is OCSP stapling. Here's how this problem is solved for OCSP. In the case of ECH, the solution would look like this:

#if (!defined OPENSSL_NO_ECH && defined SSL_OP_ECH_GREASE) ... #endif

Now this brings up another problem. In your change there are multiple conditional blocks with #ifndef OPENSSL_NO_ECH and using the condition above would significantly complicate the code, especially if we add BoringSSL ECH at some point. The solution is to minimize the number of those #if's. And again, stapling is a good example. What we need to do is to try to move all conditional compilation to ngx_event_openssl.c. I will comment the code below for more details.

Ok, aiming for that now.

As I mentioned before, I don't think it's a good idea to define/redefine OpenSSL macros like OPENSSL_NO_ECH in nginx code. I suggest using the condition #if (!defined OPENSSL_NO_ECH && defined SSL_OP_ECH_GREASE) instead and regrouping the code in a way that reduces the number of such ifs. I think 2 ifs should be enough, one for the functions and one for the variables.

Ok, tried to do that better now - should we aim for setting an NGX_ECH from auto/lib/openssl/conf or something though? Not sure if that'd be better. See what you think when I push the next rev.

src/event/ngx_event_openssl.h

src/http/modules/ngx_http_log_module.c

src/http/modules/ngx_http_ssl_module.c

arut · 2025-10-09T13:21:38Z

src/http/modules/ngx_http_ssl_module.c

+    { ngx_string("ssl_ech_inner_sni"), NULL, ngx_http_ssl_variable,
+      (uintptr_t) ngx_ssl_get_ech_inner_sni, NGX_HTTP_VAR_CHANGEABLE, 0 },
+    { ngx_string("ssl_ech_outer_sni"), NULL, ngx_http_ssl_variable,
+      (uintptr_t) ngx_ssl_get_ech_outer_sni, NGX_HTTP_VAR_CHANGEABLE, 0 },


I"m not sure we need both of them. The outer SNI does make sense. But I'm not sure we need the inner one. The outer one can be called ssl_ech_server_name, similar to ssl_server_name we already have, which is the SNI value (and is the inner SNI for ECH).

fair enough that we don't really need ssl_ech_inner_sni but I'd argue to keep the name for the outer - ssl_ech_server_name would easily confuse as to whether it means inner or outer.

In any case, we don't use the term sni in variables and directives. It's always $ssl_server_name, $ssl_preread_server_name, proxy_ssl_server_name. Let's try ssl_ech_outer_server_name at least.

ack ssl_ech_outer_server_name it is. I didn't change the function name but can if you prefer. (For now the function is still ngx_ssl_get_ech_outer_sni().

src/event/ngx_event_openssl.c

arut · 2025-10-09T13:44:30Z

src/http/modules/ngx_http_ssl_module.c


+#ifndef OPENSSL_NO_ECH
+    { ngx_string("ssl_echkeydir"),
+      NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1,


I think this directive should be allowed at the server level as well. Please add NGX_HTTP_SRV_CONF:

NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_CONF_FLAG,

I'll argue (a bit) against that - it seems simpler and better if ECH is inherently set up/available for all virtual servers rather than for some, and there's no real cost to doing so I think. I don't see why you'd want ECH to be available only for some virtual servers and not others?

I don't think we need to force it for all servers. And anyway, you are already using the server configuration (ngx_stream_ssl_srv_conf_t) which means it's already per-server. And there's no cost in enabling this directive at the server level as well. I'm not advocating for removing the main scope btw, let's just add the server scope in addition to it.

sure - did that, hopefully correctly:-)

src/event/ngx_event_openssl.c

ECH-build.md

sftcd · 2025-10-10T08:09:25Z

Those all look sensible, thanks. Will get at 'em shortly but travelling today.

arut · 2025-10-10T16:43:26Z

Those all look sensible, thanks. Will get at 'em shortly but travelling today.

Also please rebase the code to the recent master. We had some SNI-related changes there recently.

github-actions · 2025-10-12T19:09:00Z

✅ All required contributors have signed the F5 CLA for this PR. Thank you!
_{Posted by the CLA Assistant Lite bot.}

sftcd · 2025-10-12T19:13:29Z

I have hereby read the F5 CLA and agree to its terms

sftcd · 2025-10-12T19:13:47Z

recheck

sftcd · 2025-10-12T19:17:09Z

The push above fixes the things requested this week except:

yet to look at the stream module
didn't make the change away from configuring a directory name
kept the variable as ssl_ech_outer_sni
configured ECH PEM files are still server wide, rather than per virtual server

I'll take a look at 1 above and we can discuss 2/3/4. First though, I'm gonna do a bit of playing with boringssl to see if the work the lighttpd maintainer did translates over as easily as I think it should.

sftcd · 2025-10-12T22:38:31Z

The push above is a version that seems to work (with limited localhost tests) with BoringSSL. Likely would need more work, but passes those localhost tests ok (with real or GREASEd ECH and an HRR case).

sftcd · 2025-10-13T15:45:35Z

The push above tries to add ECH to the stream module. Not that sure I've done that correctly, but it works in local testing.

sftcd · 2025-10-28T18:20:31Z

@arut just checking where we're at (I have to do some project-internal reporting for month end:-) - do you need me to do anything to progress things?

arut · 2025-10-30T11:32:45Z

@arut just checking where we're at (I have to do some project-internal reporting for month end:-) - do you need me to do anything to progress things?

I'm getting back to this now. I've been busy with the release.

@sftcd, any news about split mode? Feels like split mode is a bigger feature than shared mode.

sftcd · 2025-10-30T11:53:03Z

@sftcd, any news about split mode? Feels like split mode is a bigger feature than shared mode.

Interesting! A lot of people have said to me that they don't find split-mode so interesting due to how they terminate TLS, but OTOH, we did a small trial with the Wikimedia foundation and they'd have loved to use split-mode if they had someone to be the client-facing server for them. (Finding such a someone turns out not that easy.)

Anyway, yes, I have a split-mode implementation for nginx, (and also for haproxy) using the stream module, but that needs an API that hasn't yet been agreed with the OpenSSL maintainers and that isn't in the OpenSSL feature branch. You'd have to build against a personal (or DEfO-project-specific) fork of OpenSSL rather than the OpenSSL project's ECH feature branch.

The split-mode API required is an oddball thing from the POV of the OpenSSL library - you need to use an SSL_CTX (as you don't have an SSL * connection), and the outerClientHello to get back the innerClientHello (with a bit more stuff for handling HRR to make it harder) and none of those are the sort of thing current APIs do.

You can see the DEfO-project fork's split-mode API here and the core of how we use that in nginx here. I'd not be surprised if the latter needed changes btw:-)

IIUC split-mode is also not yet supported by other TLS libraries.

So I think the way forward would be to handle ECH shared-mode first, then later to add split-mode in a separate PR once we've agreed a split-mode API with OpenSSL maintainers. (Once I've gotten that API agreed with OpenSSL maintainers I do plan to raise corresponding PRs with TLS server projects.)

arut · 2025-10-30T15:04:24Z

@sftcd apparently there's a conflict with the latest changes in nginx, probably because of $ssl_sigalg. I'm pretty sure it's something trivial. Please rebase onto the latest master branch again.

arut · 2025-10-30T15:14:28Z

src/http/modules/ngx_http_ssl_module.c

    { ngx_string("ssl_ech_status"), NULL, ngx_http_ssl_variable,
      (uintptr_t) ngx_ssl_get_ech_status, NGX_HTTP_VAR_CHANGEABLE, 0 },
-    { ngx_string("ssl_ech_inner_sni"), NULL, ngx_http_ssl_variable,
-      (uintptr_t) ngx_ssl_get_ech_inner_sni, NGX_HTTP_VAR_CHANGEABLE, 0 },
+
    { ngx_string("ssl_ech_outer_sni"), NULL, ngx_http_ssl_variable,
      (uintptr_t) ngx_ssl_get_ech_outer_sni, NGX_HTTP_VAR_CHANGEABLE, 0 },


I suggest moving these variables down somewhere between ssl_alpn_protocol and ssl_client_cert. Here the ech variables split ssl_curve and ssl_curves. Same for Stream.

The same applies to the order of function declarations in src/event/ngx_event_openssl.h and implementations in src/event/ngx_event_openssl.c.

OK, think I did that.

arut · 2025-10-30T15:17:35Z

src/event/ngx_event_openssl.h

 ngx_int_t ngx_ssl_create_connection(ngx_ssl_t *ssl, ngx_connection_t *c,
    ngx_uint_t flags);
-#ifndef OPENSSL_NO_ECH
 ngx_int_t ngx_ssl_echkeydir(ngx_conf_t *cf, ngx_ssl_t *ssl, ngx_str_t *dir);


This is not the best place for this declaration, next to ngx_ssl_create_connection() which is a runtime function as opposed to ngx_ssl_echkeydir() which is a config-time function . Please move it to other similar declarations, for example after ngx_ssl_dhparam(). Same for the implementation.

also tried to do that

arut · 2025-10-30T15:23:31Z

@sftcd, any news about split mode? Feels like split mode is a bigger feature than shared mode.

Interesting! A lot of people have said to me that they don't find split-mode so interesting due to how they terminate TLS, but OTOH, we did a small trial with the Wikimedia foundation and they'd have loved to use split-mode if they had someone to be the client-facing server for them. (Finding such a someone turns out not that easy.)

Anyway, yes, I have a split-mode implementation for nginx, (and also for haproxy) using the stream module, but that needs an API that hasn't yet been agreed with the OpenSSL maintainers and that isn't in the OpenSSL feature branch. You'd have to build against a personal (or DEfO-project-specific) fork of OpenSSL rather than the OpenSSL project's ECH feature branch.

The split-mode API required is an oddball thing from the POV of the OpenSSL library - you need to use an SSL_CTX (as you don't have an SSL * connection), and the outerClientHello to get back the innerClientHello (with a bit more stuff for handling HRR to make it harder) and none of those are the sort of thing current APIs do.

You can see the DEfO-project fork's split-mode API here and the core of how we use that in nginx here. I'd not be surprised if the latter needed changes btw:-)

IIUC split-mode is also not yet supported by other TLS libraries.

So I think the way forward would be to handle ECH shared-mode first, then later to add split-mode in a separate PR once we've agreed a split-mode API with OpenSSL maintainers. (Once I've gotten that API agreed with OpenSSL maintainers I do plan to raise corresponding PRs with TLS server projects.)

Thanks for details. Please keep us updated regarding the progress in OpenSSL.

sftcd · 2025-10-30T18:00:23Z

@sftcd apparently there's a conflict with the latest changes in nginx, probably because of $ssl_sigalg. I'm pretty sure it's something trivial. Please rebase onto the latest master branch again.

Just rebased, no other changes yet.

sftcd · 2025-10-31T15:21:27Z

Ok pushed a version that tries to address all yesterday's comments unless I missed some;-( I'd say there's likely still more to be done too. Given there's a lot of changes (moving chunks of code about) if you'd like me to squash the commits, just say. (I'll be travelling tomorrow to the IETF meeting so might be slower responding for the next week.)

arut · 2025-11-02T15:33:29Z

@sftcd, any news about split mode? Feels like split mode is a bigger feature than shared mode.

Interesting! A lot of people have said to me that they don't find split-mode so interesting due to how they terminate TLS, but OTOH, we did a small trial with the Wikimedia foundation and they'd have loved to use split-mode if they had someone to be the client-facing server for them. (Finding such a someone turns out not that easy.)

Anyway, yes, I have a split-mode implementation for nginx, (and also for haproxy) using the stream module, but that needs an API that hasn't yet been agreed with the OpenSSL maintainers and that isn't in the OpenSSL feature branch. You'd have to build against a personal (or DEfO-project-specific) fork of OpenSSL rather than the OpenSSL project's ECH feature branch.

The split-mode API required is an oddball thing from the POV of the OpenSSL library - you need to use an SSL_CTX (as you don't have an SSL * connection), and the outerClientHello to get back the innerClientHello (with a bit more stuff for handling HRR to make it harder) and none of those are the sort of thing current APIs do.

You can see the DEfO-project fork's split-mode API here and the core of how we use that in nginx here. I'd not be surprised if the latter needed changes btw:-)

IIUC split-mode is also not yet supported by other TLS libraries.

So I think the way forward would be to handle ECH shared-mode first, then later to add split-mode in a separate PR once we've agreed a split-mode API with OpenSSL maintainers. (Once I've gotten that API agreed with OpenSSL maintainers I do plan to raise corresponding PRs with TLS server projects.)

I looked at the split ech nginx implementation and I have one question. Obviously it's too early, but maybe discussing this now will help to come up with the right API.

The question is, how is config retry supposed to work? I only see one function SSL_CTX_ech_raw_decrypt() which extracts inner CH from outer CH. But what if this is not possible and nginx needs to send the current ech config? Obviously we need an SSL connection for this. But at the same time we need to feed the (already read) outer SNI to that connection and that's not something that can be easily done. So probably it's easier to start with an SSL connection anyway.

sftcd · 2025-11-02T15:55:15Z

The question is, how is config retry supposed to work? I only see one function SSL_CTX_ech_raw_decrypt() which extracts inner CH from outer CH. But what if this is not possible and nginx needs to send the current ech config? Obviously we need an SSL connection for this. But at the same time we need to feed the (already read) outer SNI to that connection and that's not something that can be easily done. So probably it's easier to start with an SSL connection anyway.

What I've done for split-mode is setup a virtual host for the public_name (or backend in haproxy terms) where connections will land if SSL_CTX_ech_raw_decrypt() fails and if the client's error was to use the wrong ECHConfig then that virtual host will handle sending back the retry_configs.

sftcd · 2025-11-02T15:56:55Z

I looked at the split ech nginx implementation and I have one question. Obviously it's too early, but maybe discussing this now will help to come up with the right API.

Forgot to say: thanks for taking a look, and if it turns out that the current API looks good to you, or we fix it to make that true, then that may well help when discussing split-mode with the OpenSSL maintainers... so thanks again:-)

arut · 2025-11-02T18:24:12Z

The question is, how is config retry supposed to work? I only see one function SSL_CTX_ech_raw_decrypt() which extracts inner CH from outer CH. But what if this is not possible and nginx needs to send the current ech config? Obviously we need an SSL connection for this. But at the same time we need to feed the (already read) outer SNI to that connection and that's not something that can be easily done. So probably it's easier to start with an SSL connection anyway.

What I've done for split-mode is setup a virtual host for the public_name (or backend in haproxy terms) where connections will land if SSL_CTX_ech_raw_decrypt() fails and if the client's error was to use the wrong ECHConfig then that virtual host will handle sending back the retry_configs.

But to do this, you need to shove the outer CH back into the socket, don't you? Or you do it on a remote server after proxying?

sftcd · 2025-11-02T19:33:16Z

But to do this, you need to shove the outer CH back into the socket, don't you? Or you do it on a remote server after proxying?

Ah, no - if the split-mode decrypt fails, the original (outer) CH is just processed as normal, so the connection will go to the server for the public_name however that's usually handled.

Not sure if it'll help or be more confusing but this config is our test case - after ECH decryption is attempted, if it worked, we'll route based on the inner SNI to foo.example.com, if ECH decryption failed, we route to example.com's server. (Note that that config doesn't take into account changes resulting from this PR as we've not updated that build for those changes yet.)

arut · 2025-11-03T05:04:16Z

But to do this, you need to shove the outer CH back into the socket, don't you? Or you do it on a remote server after proxying?

Ah, no - if the split-mode decrypt fails, the original (outer) CH is just processed as normal, so the connection will go to the server for the public_name however that's usually handled.

OK, so the outer CH is proxied in this case and then the backend will do the retry config. My point is, it would be more efficient to handle it at the proxy though. Otherwise the backend needs the public certificate/key to do the retry config, which looks completely unnecessary.

arut · 2025-11-03T15:06:16Z

Ok pushed a version that tries to address all yesterday's comments unless I missed some;-( I'd say there's likely still more to be done too. Given there's a lot of changes (moving chunks of code about) if you'd like me to squash the commits, just say. (I'll be travelling tomorrow to the IETF meeting so might be slower responding for the next week.)

In no particular order:

As we discussed before, let's move the BoringSSL part to a separate PR. I'm still a little confused here, since BoringSSL cannot generate such PEM files, how much value is in reading them with BoringSSL. The main thing is it's possible to do in a consistent way with the current BoringSSL API.
ssl_echfile - I'm not sure about the "file" part, but anyway I think we need a _ there: ssl_ech_file, a good example is ssl_ecdh_curve. Regaring the "file", sounds too general to me, but there seems to be no good term for this (echconfig+private_key) and the draft does not provide one. On the other hand, we do have ssl_password_file and ssl_stapling_file. What term do other projects use?
There should be multiple directives allowed, each of them receiving a file name. See ssl_session_ticket_key: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_session_ticket_key. The implementation should also be similar.
While I did suggest file patterns as an option for future, I don't think we need it now. Other similar directives do not currently take patterns. This may or may not came as a separate change. And the "glob" expansion will be in in http/stream code, not in ssl code. The pattern will just expand to multiple files, like a shortcut for multiple directives. And if we do it for ech, but don't do it for other directives (like ssl_session_ticket_key, or maybe even ssl_certificate/ssl_certificate_key) this will be inconsistent. This needs to be done consistently across nginx, if we decide to do it (which we have not yet decided).

We should end up with a very simple patch at this point.

Regarding the squash, yes please do it.

Have a good trip!

sftcd · 2025-11-07T15:48:22Z

@Maryna-f5 - if that milestone has a date that it'd be useful for me to understand be good to know that so I don't miss it. (I'm travelling home today so will be back at this once I get there.)

Maryna-f5 · 2025-11-07T16:49:21Z

Hi, @sftcd,

Thank you for your contribution!
We are planning to release 1.29.4 milestone on December 9, 2025.

sftcd · 2025-11-10T13:47:12Z

Gonna start doing that squash-commits and splitting out the boring stuff shortly. Will post on here when things are ready for another review, as I might temporarily break something in the PR branch along the way:-)

sftcd · 2025-11-10T22:01:48Z

OK, I think the push above is the more minimal PR asked-for. It squashed commits, allows multiple ssl_echfile directives, and doesn't include BoringSSL nor globbing support. (I'll make a PR with both of those too in a bit.) I also rebased and added a small new thing - only the first named file from an ssl_echfile will be used in an ECH retry-config during the ECH fallback.

sftcd · 2025-11-10T22:06:51Z

The more complete PR (with boriing support and globbing) is #973. A couple of notes on that:

Yes, BoringSSL doesn't produce the ECH PEM files, but you can use boring's bssl command with a small script wrapper to make them .(Here's one)
Given BoringSSL "releases" have ECH by default but OpenSSL doesn't yet include it in releases, that may argue to include boring support as soon as possible.
You'll notice that there's a small bit of code in this PR now that's done to make ECH shared-mode with both BoringSSL and globbing #973 easier (e.g. an abstract ECH store void *). If you'd prefer not have that, that's fine.
If you wanted, I'm fine with separating out the BoringSSL support from the globbing stuff, but I get the impression we'll be parking ECH shared-mode with both BoringSSL and globbing #973 for a while (which is fine if that's what's preferred) so I've left 'em together for now as the easier thing to do:-)

sftcd · 2025-11-11T12:17:52Z

oops, I notice I forgot to do s/ssl_echfile/ssl_ech_file/ in that push, will do next go around if one's needed (probably:-)

arut

Generally, the change looks mostly ok. I suggest some cleanup and simplification.

I left some comments, mostly style-related. Generally, I suggest looking at the code in src/event/ngx_event_openssl.c and copying the style.

arut · 2025-11-11T15:45:05Z

src/event/ngx_event_openssl.c

+/* check defines from <openssl/ssl.h> for OpenSSL ECH support */
+#if !defined(OPENSSL_NO_ECH) && defined(SSL_OP_ECH_GREASE)
+#define NGX_ECH
+#endif
+


I looked at this again and I think we can eliminate NGX_ECH and use #ifdef SSL_OP_ECH_GREASE instead, just like we do in other places. A good example is #ifdef SSL_READ_EARLY_DATA_SUCCESS.

For BoringSSL we'll choose another macro. A good example is ngx_ssl_early_data() where we use different macros for OpenSSL and BoringSSL.

Sure. Had to re-check how that connected with OPENSSL_NO_ECH but as of now SSL_OP_ECH_GREASE is only defined if OPENSSL_NO_ECH is not. That could change though to avoid possible conflicts since SSL_OP_ECH_GREASE is an index for a bit-mask, which is why I did it as above. But, in any case, you'd get a compiler or linker error I guess, so I'll do the change requested.

arut · 2025-11-11T15:49:59Z

src/event/ngx_event_openssl.c

+static ngx_int_t
+ngx_ssl_load_one_echkey(ngx_ssl_t *ssl, ngx_str_t *filename, void *echstore, 
+                        int for_retry)
+{
+    BIO             *in;
+    ngx_int_t        rv;
+    OSSL_ECHSTORE   *es;
+
+    rv = NGX_ERROR;
+    in = BIO_new_file((char *)filename->data, "r");
+    if (in == NULL)
+        return NGX_ERROR;
+    es = (OSSL_ECHSTORE*) echstore;
+    if (1 == OSSL_ECHSTORE_read_pem(es, in, for_retry))
+        rv = NGX_OK;
+    BIO_free_all(in);
+    return rv;
+}


I suggest inlining this function into ngx_ssl_echfiles().

arut · 2025-11-11T15:50:44Z

src/event/ngx_event_openssl.c

+        if (somekeyworked == 0) {
+            ngx_ssl_error(NGX_LOG_EMERG, ssl->log, 0,
+                          "ngx_ssl_echfiles loaded no keys but ECH configured");
+            return NGX_ERROR;
+        }


Looks like dead code. I suggest removing it, as well as the somekeyworked variable.

arut · 2025-11-11T15:51:32Z

src/event/ngx_event_openssl.c

+        }
+
+        /* subsequent times around the loop not returned for retries */
+        for_retry = OSSL_ECH_NO_RETRY;


We can derive this value from i. I suggest removing the for_retry variable.

OK, though for me that's less clear for a reader but I added a comment.

arut · 2025-11-11T15:52:56Z

src/event/ngx_event_openssl.c

+        for_retry = OSSL_ECH_NO_RETRY;
+    }
+
+    if (OSSL_ECHSTORE_num_keys(es, &numkeys) != 1


Why request the number of keys? If the array exists, the number of files should be >= 1, so as the number of keys. I suggest removing this.

It is allowed to have PEM files with just ECHConfigList values and no private key (the current spec is for zero or one private keys per file plus one ECHConfigList), so we need to check we have some private key before ECH can work. I cleaned up a bit and added a comment to explain.

arut · 2025-11-11T16:07:39Z

src/event/ngx_event_openssl.c

+#endif
+
+#endif
+    return NGX_OK;


There needs to be a default s->len = 0 in this function.

arut · 2025-11-11T16:09:09Z

src/event/ngx_event_openssl.c

+        ngx_str_set(s, "FAILED");
+        break;
+    case SSL_ECH_STATUS_BAD_NAME:
+        ngx_str_set(s, "WORKED_BAD_NAME");


Why not just BAD_NAME?

That's really for completeness and would only happen on a client. If at some point, you extend nginx as a proxy that terminates TLS inbound and sets up TLS outbound with ECH used for the latter then it could be a status that's returned. It could be taken out for now if desired but might get forgotten later. As to the string, I figured just BAD_NAME might be more easily confused with the ECH fallback via retry-configs, but can change to that if you prefer.

arut · 2025-11-11T16:21:20Z

src/event/ngx_event_openssl.c

+    case SSL_ECH_STATUS_BACKEND:
+        ngx_str_set(s, "INNER");
+        break;


Could you explain what this code means? And what's the difference between this and SUCCESS?

In ECH split-more the backend server will get a ClientHello that does contain an ECH extension but with the 'inner' value set. Even though we're not supporting split-more as a client-facing server (i.e. front-end) yet, we might be a backend behind some who is an ECH front-end, so this would report on that situation. And it's not ECH 'SUCCESS' since we didn't ECH decrypt, but just got presented with a ClientHelllo containing an ECH extension set to 'inner'.

arut · 2025-11-11T16:30:02Z

src/event/ngx_event_openssl.c

+    if (in == NULL)
+        return NGX_ERROR;
+    es = (OSSL_ECHSTORE*) echstore;
+    if (1 == OSSL_ECHSTORE_read_pem(es, in, for_retry))


Style: we prefer to use if (fun() == 1) {..} instead of if (1 == fun()) {..}.

arut · 2025-11-11T16:48:41Z

src/stream/ngx_stream_ssl_module.c

      offsetof(ngx_stream_ssl_srv_conf_t, dhparam),
      NULL },

+    { ngx_string("ssl_echfile"),


ssl_ech_file? Same for the field and function names.

think I got all those now (bit of testing then I'll do a push)

sftcd · 2025-11-11T22:33:06Z

Push above tries to handle all today's @arut comments.

sftcd mentioned this pull request Aug 18, 2025

OpenSSL ECH integration apache/httpd#551

Closed

sftcd mentioned this pull request Aug 27, 2025

ECH (Encrypted client hello) support haproxy/haproxy#1924

Closed

ammnt mentioned this pull request Aug 28, 2025

Encrypted Client Hello webserver-llc/angie#58

Closed

yaroslavros reviewed Sep 6, 2025

View reviewed changes

arut reviewed Sep 25, 2025

View reviewed changes

arut requested changes Oct 9, 2025

View reviewed changes

sftcd force-pushed the ECH-shared branch from 9ba3676 to 170eb28 Compare October 12, 2025 19:08

brianshea2 mentioned this pull request Oct 29, 2025

[Request] Add checks for ECH and WebRTC leaks brianshea2/addr.tools#39

Closed

arut reviewed Oct 30, 2025

View reviewed changes

sftcd force-pushed the ECH-shared branch from 812e356 to eaa40dc Compare October 30, 2025 17:59

Maryna-f5 added this to the nginx-1.29.4 milestone Nov 6, 2025

Maryna-f5 added the feature label Nov 6, 2025

Add basic ECH shared-mode via OpenSSL

cf51799

sftcd force-pushed the ECH-shared branch from 4d59a3a to cf51799 Compare November 10, 2025 21:59

sftcd mentioned this pull request Nov 10, 2025

ECH shared-mode with both BoringSSL and globbing #973

Open

arut requested changes Nov 11, 2025

View reviewed changes

fixup! Add basic ECH shared-mode via OpenSSL

985dcd0

OpenSSL ECH integration #840

Are you sure you want to change the base?

OpenSSL ECH integration #840

Conversation

sftcd commented Aug 12, 2025

Proposed changes

Uh oh!

yaroslavros commented Sep 6, 2025

Uh oh!

sftcd commented Sep 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yaroslavros commented Sep 6, 2025

Uh oh!

sftcd commented Sep 6, 2025

Uh oh!

sftcd commented Sep 12, 2025

Uh oh!

arut commented Sep 25, 2025

Uh oh!

arut commented Sep 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sftcd commented Sep 25, 2025

Uh oh!

sftcd commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sftcd commented Sep 26, 2025

Uh oh!

sftcd commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arut left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sftcd commented Oct 10, 2025

sftcd commented Sep 26, 2025 •

edited

Loading

sftcd commented Sep 26, 2025 •

edited

Loading

github-actions bot commented Oct 12, 2025 •

edited

Loading

sftcd commented Oct 12, 2025 •

edited

Loading

arut commented Nov 2, 2025 •

edited

Loading