This problem occurred with tlsv1.3 and tlsv1.2 specified in versions. So I changed it to tlsv1.2 only and the problem was solved.

Perhaps the client specifies both TLS 1.3 and 1.2, but the signature algorithm selection process on the client side is TLS 1.3, even though the server side has selected TLS 1.2, and the rsa_pkcs1 process I think they are being discouraged.

Describe the bug

When using TLS 1.3 and TLS 1.2 with ssl, with mTLS (client authentication),
Certificate Request message from the server contains rsa_pkcs1 as the signature algorithm,
However, the certificate signed with rsa_pkcs1 is not used and an empty Certificate message is sent.

I am thinking that the following part may be wrong, but I have not found the problem part.
https://github.com/erlang/otp/blob/master/lib/ssl/src/ssl_handshake.erl#L1668-L1748

To Reproduce

The client passes a minimum of options to ssl:connect. The certificate is signed with sha384WithRSAEncryption.

-module(mtls).

-feature(maybe_expr, enable).

-export([run/0]).

run() ->
    maybe
        {ok, _Started} ?= application:ensure_all_started(ssl),
        TlsOpts = [{versions, ['tlsv1.3', 'tlsv1.2']},
                   {verify, verify_none},
                   {certfile, <<"cert.pem">>},
                   {keyfile, <<"key.pem">>}],
        {ok, _Socket} ?= ssl:connect("localhost", 4433, TlsOpts, 5000),
        ok
    else
        Reason ->
            io:format("Error: ~p~n", [Reason]),
            error
    end.

Server is OpenSSL with mTLS enabled, TLS 1.2 enforced,
The server uses OpenSSL with mTLS enabled, TLS 1.2 enforced, and a narrower signature algorithm.
The same certificate is used for both client and server for verification purposes.

$ openssl s_server -accept 4433 -cert cert.pem -key key.pem -Verify 1 -tls1_2 -cipher 'ECDHE-RSA-AES128-GCM-SHA256' -client_sigalgs rsa_pkcs1_sha384

When executed, the following error occurs.

1> mtls:run().
=NOTICE REPORT==== 22-Dec-2023::16:03:22.781490 ===
TLS client: In state cipher received SERVER ALERT: Fatal - Handshake Failure

Error: {error,
           {tls_alert,
               {handshake_failure,
                   "TLS client: In state cipher received SERVER ALERT: Fatal - Handshake Failure\n"}}}
error

Expected behavior

When TLS 1.3 and TLS 1.2 are specified in versions and mTLS is used, when a Certificate Request message is sent with rsa_pkcs1 included, a certificate signed by rsa_pkcs1 is included in the Certificate message

Affected versions

OTP-26.2.1

Additional context

The verification certificates cert.pem and key.pem used are on the following Gist.

https://gist.github.com/voluntas/47b95f54069f9728189041583c20aab7

Incidentally, it works if the signature algorithm includes rsa_pss_rsae; we found this problem because some servers did not include rsa_pss_rsae in the Certificate Request message.

Only rsa_pkcs1_sha384 is included in Certificate Request in TLS 1.2

Client Certificate is empty in TLS 1.2