Background and motivation

See full discussion at dotnet/aspnetcore#58560

As part of TLS handshake the certificate private key is used to compute a signature over the handshake to ensure TLS properties. Normally, the type of the private key inside the certificate directly identifies the signature algorithm that can be used, but the RSA algorithm has two padding modes which can be used with the same certificate.

The linked issue above mentions a specific problem where particular RSA padding mode needs to be disabled to unblock users: some older TPM modules do not support the rsa_pss_rsae_* group of algorithms which causes the handshake failure. The ability to explicitly opt out of such padding mode allows the implementation to choose a different (=supported) signature algorithm.

WIP implementation available at #114741

API Proposal

namespace System.Net.Security;

public class SslClientAuthenticationOptions
{
+       public bool AllowRsaPssPad { get; set; } = true;
+       public bool AllowRsaPkcsPad { get; set; } = true;
}

public class SslServerAuthenticationOptions
{
+       public bool AllowRsaPssPad { get; set; } = true;
+       public bool AllowRsaPkcsPad { get; set; } = true;
}

AllowRsaPssPad controls rsa_pss_* algorithms, AllowRsaPkcsPad controls rsa_pkcs1_* algorithms.

API Usage

sslStream.AuthenticateAsServer(new SslServerAuthenticationOptions
{
    // ....
    AllowRsaPssPad = false,
}

The snippet above causes the following:

• The server will not chose rsa_pss_* signature algorithm to sign the handshake (CertificateVerify message)
• The server will not advertise willingness to accept rsa_pss_* when requesting client certificate (in CertificateRequest TLS message)

For client, the behavior is similar: the client will not advertise rsa_pss_* in ClientHello, and will if server requests a client certificate, will not choose rsa_pss_* to sign the handshake.

Alternative Designs

instead of two boolean fields, a flag enum field may be appropriate, however, RSASignaturePaddingMode already exists so this would introduce a similar, non-compatible type.

Risks

The API as proposed is implementable for Windows and Linux. It is unknown whether there is a viable way to implement this on OSX and mobile platforms.

Implementation hurdles on Linux

On Linux specifically, the underlying OpenSSL platform provides only the "allowlist" approach. By default no explicit allowed cipher list is specified and the list is left up to the library (and global openssl.cnf configuration). If any of the new propeties is set to false (non-default), we need to

• detect the default allowed signature algorithms
• remove the ones we want to disable

Due to implementation difficulties, it is necessary to embed the knowledge of existing signature algorithms in the System.Net.Security internals, and if ever new signature algorithms become standard (e.g. Post Quantum Cryptography), we would need to update the internal list of known sigalgs (and backport the change to servicing .NET releases), or ignore all new sigalgs introduced in the future (the less preferred solution). Note that this would affect only users who explicitly opt-out of any of the padding methods.