Draft: Generalize API for extensible non-X.509 support #2560

rrauch · 2025-07-14T12:23:48Z

What

This draft PR introduces an approach for cleaner, more extensible support of non-X.509 identity schemes, such as Raw Public Keys. The focus is on a long-term design that better represents the realities of modern TLS. It also solves issue #2258.

Why

Historically, TLS and X.509 certificates were tightly coupled, both by convention and implementation. However, the TLS protocol has been evolving towards a more flexible design. RFC 7250 added support for Raw Public Keys (RPK) to TLS 1.2, and RFC 8446 (TLS 1.3) went further by making certificate_type negotiation a standard part of the handshake.

The specification is extensible, currently allowing:

X.509 (certificate type id 0)
RPK (certificate type id 2)
Custom/private schemes (certificate type id range 224-255)

(Note: OpenPGP, which used certificate type id 19, was specified in RFC 6091, but its support was removed in RFC 8446 due to a lack of adoption.)

Rustls' initial RPK support (PR #2062) minimized changes to the public API by reusing existing types (CertificateDer to hold RPKs) and traits (CertVerifier). This required adding specialized methods like requires_raw_public_keys and hardcoded logic to the handshake to differentiate between schemes.

While this approach reduced the scope of the PR and preserved backward compatibility, it limits how cleanly Rustls can support schemes beyond X.509, as highlighted in discussions #2258 and #2266. The CertVerifier traits, designed exclusively for X.509, now have potentially invalid states and are awkward to use for other identity types.

Furthermore, the current design does not scale to support future schemes and does not support custom/private types at all. For me, enabling custom schemes is the primary motivation for this PR.

How

Introduced Identity Abstraction: This PR introduces a new TlsIdentity trait to decouple the API from X.509 certificates. Implementations for X.509 and RPK are included. A peer's identity is accessible via peer_identity(), while peer_certificates() is kept for backward compatibility. Convenience methods like Identity::as_certificates and Identity::as_public_key are provided, with Identity::as_any available for custom types.
New Verifier/Resolver Traits: New, scheme-neutral traits have been introduced: ServerIdVerifier, ClientIdVerifier, ResolvesServerIdentity, and ResolvesClientIdentity. Implementations are only required for new or custom schemes. This PR includes implementations for X.509 (wrapping the existing CertVerifier traits for compatibility) and new, purpose-built RpkVerifier/RpkResolver traits for RPK users.
Dynamic Handshake Logic: Hardcoded handshake logic has been replaced with a dynamic approach that delegates negotiation to the new verifier/resolver traits. X.509-specific code has been moved into its respective trait implementation, making the main code paths identity-scheme neutral.
Support for Custom Types: The CertificateType enum was extended to support the custom/private use range (224-255). Basic range support was added to the enum_builder.
Terminology Update: Throughout the codebase, certificate/cert has been replaced with identity/id where appropriate to reflect the new abstractions.

Impact

The changes are necessarily invasive but establish a foundation for long-term API evolution. The number of changed LOC is substantial, though many are simple renames.

Code Changes

The most meaningful changes are in:

identity.rs: Contains the main Identity type, the TlsIdentity & IdentitySigner traits and their respective implementations.
verify.rs: Adds the new ServerIdVerifier & ClientIdVerifier traits, compatibility wrappers for X.509, and the new ServerRpkVerifier & ClientRpkVerifier traits.
server_conn.rs & client_conn.rs: Adds new identity resolvers and public setters for using non-X.509 schemes.
common_state.rs: Adds the peer_identity() method and fields to store the negotiated certificate types.
Handshake logic: Updated to use the new dynamic negotiation. The changes are mostly related to TLS 1.3, TLS 1.2 remains limited to X.509 certificate support.
builder.rs: Adds with_*_rpk_resolver and with_*_identity_resolver methods to supplement existing ones.

Performance

Performance should be unaffected. Any significant regression should be considered a bug.

Backward Compatibility

Although the changes are substantial, care has been taken to minimize breakage.

RPK Users: This is a breaking change. Users must migrate to the new, purpose-built RPK traits (e.g., ClientRpkVerifier instead of ClientCertVerifier) and use the new with_rpk_* configuration methods. The peer's key should be accessed via peer_identity().as_public_key(). These changes result in a cleaner, more correct API for RPK usage.
Direct Config Field Access: For users who directly access the public ClientConfig::client_auth_cert_resolver or ServerConfig::cert_resolver fields, 100% backward compatibility cannot be maintained, as these fields have been renamed and re-typed. Affected users must use the similarly named methods instead (e.g. change from config.cert_resolver = X; to config.cert_resolver(X);.
Note: Users who rely on the builder methods are not affected.

Overall, this has been an interesting endeavor. I believe these changes are needed in one form or another. Looking forward to everyone's feedback and opinions.

Replace certificate-centric architecture with Identity abstraction to cleanly support X.509 certificates, Raw Public Keys (RPK), and custom authentication types. Changes: - Add Identity concept as unified abstraction above certificate types - Refactor core logic to use Identity instead of direct certificate handling - Implement clean RPK support via new trait interfaces - Add extensibility for custom certificate/auth types - Maintain full backward compatibility for existing X.509 workflows This establishes a proper architectural foundation for diverse authentication mechanisms beyond the previous X.509-focused design.

ctz

This is a quick, partial review. I think there are some reasonable ideas in here, and I'm open to this sort of change, but one big commit makes it hard to evaluate in detail.

ctz · 2025-07-15T10:40:45Z

rustls/src/crypto/signer.rs

+}
+
+#[derive(Clone, Debug)]
+pub struct KeyPair {


I think this naming is much too generic. I am concerned that a typical user of rustls does not have a use for RPK, but this naming makes it sound enticingly useful.

Maybe we could have an rustls::rpk module (or something) collecting these things, whose doc comment says "if you stumbled upon this this isn't what you want"

ctz · 2025-07-15T10:41:57Z

rustls/src/identity.rs

+use core::fmt::Debug;
+use pki_types::{CertificateDer, SubjectPublicKeyInfoDer};
+use std::any::Any;
+use std::prelude::rust_2015::{Box, Vec};


This is odd. Use alloc::vec::Vec and alloc::boxed::Box?

ctz · 2025-07-15T10:42:57Z

rustls/src/lib.rs

-    macro_rules! trace    ( ($($tt:tt)*) => {{}} );
-    macro_rules! debug    ( ($($tt:tt)*) => {{}} );
-    macro_rules! error    ( ($($tt:tt)*) => {{}} );
-    macro_rules! _warn    ( ($($tt:tt)*) => {{}} );
+    macro_rules! trace ( ($($tt:tt)*) => {{}} );
+    macro_rules! debug ( ($($tt:tt)*) => {{}} );
+    macro_rules! error ( ($($tt:tt)*) => {{}} );
+    macro_rules! _warn ( ($($tt:tt)*) => {{}} );


unnecessary whitespace change

ctz · 2025-07-15T14:40:26Z

rustls/src/client/client_conn.rs

-use alloc::vec::Vec;
-use core::marker::PhantomData;
-use core::ops::{Deref, DerefMut};


Please run cargo +nightly fmt-unstable to sort out the import ordering/groupings.

ctz · 2025-07-15T15:07:56Z

rustls/src/client/client_conn.rs

+#[derive(Debug)]
+pub(crate) struct ResolvesClientCertCompat(Arc<dyn ResolvesClientCert>);


I appreciate the idea of this compatibility layer, but I think ResolvesClientIdentity can replace ResolvesClientCert in its entirety. (main is open for breaking changes right now, but I suspect it wasn't when you started these changes.)

ctz · 2025-07-15T15:35:46Z

rustls/src/client/tls13.rs

+        let cert_type = cx
+            .common
+            .negotiated_client_cert_type()


Generally let's not insert things into cx.common early in the handshake and then extract them later. Instead, thread these into the handshake states and expose them in the public API at exactly the same point as cx.common.peer_identity is changed.

ctz · 2025-07-15T15:38:45Z

rustls/src/msgs/base.rs

+impl<'a> From<&'a Vec<u8>> for Payload<'a> {
+    fn from(value: &'a Vec<u8>) -> Self {
+        Self::Borrowed(value.as_slice())
+    }
+}


This is weird.

ctz · 2025-07-15T15:42:22Z

rustls/src/msgs/handshake.rs

-    pub(crate) cert: CertificateDer<'a>,
-    pub(crate) extensions: CertificateExtensions<'a>,
+#[derive(Debug, Clone)]
+pub struct IdentityEntry<'a> {


These types should be named -- as far as possible -- after types in RFC8446. I suggest we keep this as CertificateEntry but make it a rust enum with RawPublicKey and X509 variants. ref: https://datatracker.ietf.org/doc/html/rfc8446#section-4.4.2

ctz · 2025-07-15T15:45:12Z

rustls/src/msgs/handshake.rs

-            Certificate(x) => x.encode(bytes),
+            Certificate(x) => Codec::encode(x, bytes),


This is weird.

ctz · 2025-07-15T15:50:32Z

rustls/src/enums.rs

+        !Ranges:
+          Custom => 224..=255,


This seems to be unused/unmotivated? FWIW this type can already represent the full range of standard values (unassigned ones via the Unknown(_) variant)

djc · 2025-07-16T07:07:29Z

Conceptually this is definitely interesting to me -- decoupling TLS and X.509 seems like a worthwhile way to further modularize rustls and make it usable across a broader range of use cases. Ideally we'd broaden the interface while trying not to meaningfully degrade the ergonomics for the vast majority that wants to just use certificates.

On the implementation side, I think this needs a lot of work to be viable. I'd suggest picking much smaller chunks to happen at a time, starting with picking client or server and working out what it needs.

Some aspects of your PR seem to be affected by LLM-assisted coding. While I'm a happy Copilot user, I'd strongly suggest writing any descriptive prose yourself and making sure to closely self-review any LLM-generated code before submitting it to us for review. We usually work with small commits exactly because they make review easier. Most of this is explained in our CONTRIBUTING.md.

rrauch added 6 commits July 10, 2025 18:39

Minor changes and oversight fixes

7f5b138

Reverted unecessary changes

1d8af7a

Fixed overlooked tests

92809cb

Import optimizations, minor changes and fixes

7164fd2

fix: adapt to upstream changes after rebase

a2ef887

rrauch marked this pull request as draft July 14, 2025 12:25

Merge branch 'main' into cleaner_non_x509_support

162fcf9

ctz reviewed Jul 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Draft: Generalize API for extensible non-X.509 support #2560

Draft: Generalize API for extensible non-X.509 support #2560

rrauch commented Jul 14, 2025

Uh oh!

ctz left a comment

Uh oh!

ctz Jul 15, 2025

Uh oh!

ctz Jul 15, 2025

Uh oh!

ctz Jul 15, 2025

Uh oh!

ctz Jul 15, 2025

Uh oh!

ctz Jul 15, 2025

Uh oh!

ctz Jul 15, 2025

Uh oh!

ctz Jul 15, 2025

Uh oh!

ctz Jul 15, 2025

Uh oh!

ctz Jul 15, 2025

Uh oh!

ctz Jul 15, 2025

Uh oh!

djc commented Jul 16, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

		#[derive(Debug)]
		pub(crate) struct ResolvesClientCertCompat(Arc<dyn ResolvesClientCert>);

		Certificate(x) => x.encode(bytes),
		Certificate(x) => Codec::encode(x, bytes),

Draft: Generalize API for extensible non-X.509 support #2560

Are you sure you want to change the base?

Draft: Generalize API for extensible non-X.509 support #2560

Conversation

rrauch commented Jul 14, 2025

What

Why

How

Impact

Code Changes

Performance

Backward Compatibility

Uh oh!

ctz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

djc commented Jul 16, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.