TLS 1.2 handshake fails with ecdsa_secp521r1_sha512 (Invalid peer certificate: BadSignature) #2477
Description
Checklist
- I've searched the issue tracker for similar bugs.
Describe the bug
When using aws-lc-rs as the cryptographic backend for TLS 1.2 connections, clients fail to complete the handshake if the server certificate uses ecdsa_secp521r1_sha512 (P-521 curve with SHA-512).
To Reproduce
1.Configure a client using aws-lc-rs as the crypto backend
2.Attempt to connect to dns.pub (DNS-over-HTTPS service)
3.Observe handshake failure with error: invalid peer certificate: BadSignature
4.Repeat identical test using ring instead - connection succeeds
I wrote a demo to verify my problem: https://github.com/Orvaxis/rutls-test
This is the result of running on a Debian 12 VPS
Rustls TLS Provider Comparison Test
Testing fixed cipher suite configuration
=== TLS Provider Performance Test ===
Target: 120.53.53.53:443
Server: dns.pub
1. AWS-LC-RS:
[AWS-LC-RS] Connecting to 120.53.53.53:443...
[AWS-LC-RS] TLS handshake successful!
[AWS-LC-RS] Failed: invalid peer certificate: BadSignature
2. Ring:
[Ring] Connecting to 120.53.53.53:443...
[Ring] TLS handshake successful!
[Ring] Response received:
HTTP/1.1 502 Bad Gateway
Server: nginx
Date: Mon, 02 Jun 2025 13:51:40 GMT
Content-Type: text/html
Content-Length: 150
Connection: close
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
[Ring] Success, time: 11.462508msApplicable Version(s)
macOS 14 x86_64
Debian12 X86_64
Expected behavior
The handshake should succeed, as P-521/SHA-512 is a valid NIST-approved ECDSA suite for TLS 1.2 (RFC 8422).
Additional context
Unlike ring (which explicitly doesn’t support P-521/SHA-512), aws-lc-rs claims support for this algorithm but fails during actual handshakes.
Packet captures confirm the handshake fails at certificate verification (BadSignature), suggesting an implementation gap in aws-lc-rs’s P-521 signature validation.