gh-135401: Test AWS-LC as a cryptography library in CI #135402

WillChilds-Klein · 2025-06-11T16:48:52Z

Notes

This PR extends multissltests.py's AbstractBuilder class to fetch AWS-LC v1.55.0 and build it using CMake and GNU make. To do this, we add cmake as a GitHub Runner dependency in .github/workflows/posix-deps-apt.sh. We also update CPython's configure and configure.ac scripts to swap out BLAKE2 (not tracked for standardization) in favor of SHA-512 when detecting libcrypto compilation compatibility for hashlib.

Finally, a new CI workflow uses this update to dynamically link AWS-LC against CPython, perform a linkage check, and run CPython's ssltests.py in CPython's public CI. This differs from AWS-LC's own CPython integration test where we statically link the CPython executable to AWS-LC.

The new CI check is not marked as "required", but if the community wants to make it "required" for future PRs that can be done by adding a list item for build-ubuntu-ssltests-awslc here.

Please feel free to file an issue with the AWS-LC team here for assistance in troubleshooting any CI failures of the new check.

Testing

New CI job completed successfully

Issue: Add ssl module test for AWS-LC #135401

This reverts commit 7d37e6a.

Misc/NEWS.d/next/Tests/2025-06-11-16-52-49.gh-issue-135401.ccMXmL.rst

Tools/ssl/multissltests.py

AA-Turner · 2025-06-11T21:40:19Z

.github/workflows/build.yml

+      with:
+        path: ./multissl/aws-lc/${{ matrix.awslc_ver }}
+        key: ${{ matrix.os }}-multissl-aws-lc-${{ matrix.awslc_ver }}
+    # TODO [childw] can we use env.* instead of env vars here?


I'd suggest for the initial version, keep as similar to the OpenSSL job/workflow, and then perhaps update both at once afterwards?

Fair enough. I'll remove the TODOs. Perhaps we can leave this comment unresolved as a reminder for me to clean up both (if tenable) if/after this PR has been merged.

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…XmL.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

Tools/ssl/multissltests.py

configure.ac

picnixz · 2025-06-13T16:56:55Z

Can you cherry-pick 8f4a0eb and make a separate PR please? TiA.

Misc/NEWS.d/next/Tests/2025-06-11-16-52-49.gh-issue-135401.ccMXmL.rst

picnixz · 2025-07-04T21:44:30Z

Tools/ssl/multissltests.py

+    if not args.openssl and not args.libressl and not args.awslc:
        args.openssl = list(OPENSSL_RECENT_VERSIONS)
        args.libressl = list(LIBRESSL_RECENT_VERSIONS)
+        args.awslc = list(AWSLC_RECENT_VERSIONS)


I don't like the rigidity of this as it scales poorly when we introduce more implementations (I don't see BoringSSL for instance). But let's keep this as is and I'll refactor this script in a separate PR.

Misc/NEWS.d/next/Tests/2025-06-11-16-52-49.gh-issue-135401.ccMXmL.rst

…XmL.rst

Misc/NEWS.d/next/Tests/2025-06-11-16-52-49.gh-issue-135401.ccMXmL.rst

Tools/ssl/multissltests.py

…XmL.rst Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

hugovk

There's a lot of duplication between the build-ubuntu-ssltests-openssl and build-ubuntu-ssltests-awslc jobs, can we use a matrix to combine them?

Something like hugovk@a3f2ba9 -> https://github.com/hugovk/cpython/actions/runs/16121165851

Tools/ssl/multissltests.py

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

WillChilds-Klein · 2025-07-07T16:02:20Z

There's a lot of duplication between the build-ubuntu-ssltests-openssl and build-ubuntu-ssltests-awslc jobs, can we use a matrix to combine them?

Something like hugovk@a3f2ba9 -> https://github.com/hugovk/cpython/actions/runs/16121165851

I think that makes a lot of sense. @AA-Turner previously suggested that I leave use OpenSSL's established CI definition patterns for this PR, then follow up with another PR to refactor CI definitions.

@hugovk @AA-Turner -- Would you prefer that I incorporate the CI refactor into this PR or a fast-follow-up PR?

picnixz · 2025-07-07T16:18:02Z

I think it's better to make it in a separate one. It'll be easier to revert if bad things happen.

hugovk

Thank you!

zware · 2025-07-11T20:30:17Z

Tools/ssl/multissltests.py

+    color=True,
+    suggest_on_error=True,


These seem to be causing us some issues:

Suggested change

color=True,

suggest_on_error=True,

**(

dict(

color=True,

suggest_on_error=True,

) if sys.version_info >= (3, 14) else {}

)

Or, to simplify:

Suggested change

color=True,

suggest_on_error=True,

Considering the comment on line 21, the latter is probably the better option.

Thanks, I'll remove the coloration directives for now.

This was suggested by @hugovk -- if he prefers, I'll revert and use @zware's kwarg-dict-unpacking approach.

Ah right, revert is the way to go, thanks.

It's because this script is run before we've built our own Python and we're using image-default one, likely version 3.13.

https://github.com/python/cpython/actions/runs/16229017511/job/45827626323?pr=135402#step:9:1

We can instead do like parser.color = True, but it can be for the refactor.

https://docs.python.org/3.14/library/argparse.html#color

Tools/ssl/multissltests.py

zware · 2025-07-11T20:35:03Z

Tools/ssl/multissltests.py

+            "-DFIPS=OFF",
+        ]
+        self._subprocess_call(cmd, cwd=cwd, env=env)
+        self._subprocess_call(["make", f"-j{self.jobs}"], cwd=cwd, env=env)


Suggested change

self._subprocess_call(["make", f"-j{self.jobs}"], cwd=cwd, env=env)

self._subprocess_call(["make", "-j{}".format(self.jobs)], cwd=cwd, env=env)

Alternatively, it's probably time that we could remove 2.7 from the list of versions we should stay compatible with.

Alternatively, it's probably time that we could remove 2.7 from the list of versions we should stay compatible with.

That's certainly a call for the maintainers, but I'm happy to help implement your decision. No concerns from AWS-LC's perspective as the earliest version we test against is 3.9.

@zware -- the 2.7 compat issues didn't show up in this PR's CI. is there some other slack/zulip/etc. channel where i can monitor secondary CI builds to get ahead of these issues during review?

I personally have been ready to drop 2.7 compatibility everywhere for over a decade now :). I have no evidence that anyone is still actually using 2.7 with this script; that note is 8 years old, from back when we were still maintaining each of the mentioned versions.

Co-authored-by: Zachary Ware <zachary.ware@gmail.com>

zware · 2025-07-11T22:22:39Z

I think we've quibbled enough on this; I'm merging it :). I'm re-running the new job one last time just to be sure, but setting automerge anyway. Since it's non-blocking anyway, we can fix up anything else in followups.

hugovk · 2025-07-12T10:13:55Z

@WillChilds-Klein Thank you very much for this!

…H-135402) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Zachary Ware <zach@python.org>

WillChilds-Klein added 5 commits June 11, 2025 16:19

First cut, minimal testing

57209fe

Various fixups, multissl.py works to build AWS-LC

8fb1016

Add linkage check

67fd836

Fix line break in configure command

4f0928b

Fix more formatting

b65d662

bedevere-app bot mentioned this pull request Jun 11, 2025

Add ssl module test for AWS-LC #135401

Closed

WillChilds-Klein and others added 9 commits June 11, 2025 16:51

Fix more formatting

6791473

📜🤖 Added by blurb_it.

269dc10

Fix all-required-greed target

cd74e2b

Fix aws-lc build dir path

24fbecf

Remove NID_blake2b512 from hashlib dependency test in ./configure

fa08737

Remove NID_blake2b512 from configure.ac

3f3a70b

Tryi setting --with-openssl-rpath in configure invocation

7d37e6a

Revert "Tryi setting --with-openssl-rpath in configure invocation"

6eb1190

This reverts commit 7d37e6a.

Guard new hashlib ctor test

8f4a0eb

AA-Turner changed the title ~~gh-135401 Add AWS-LC-backed ssl module CI job~~ gh-135401: Test AWS-LC SSL in CI Jun 11, 2025

AA-Turner reviewed Jun 11, 2025

View reviewed changes

WillChilds-Klein and others added 3 commits June 11, 2025 18:15

Update Tools/ssl/multissltests.py

7ebee26

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

Update Tools/ssl/multissltests.py

840923d

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

Update Misc/NEWS.d/next/Tests/2025-06-11-16-52-49.gh-issue-135401.ccM…

3850ba0

…XmL.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

WillChilds-Klein changed the title ~~gh-135401: Test AWS-LC SSL in CI~~ gh-135401: Test AWS-LC as a cryptography library in CI Jun 11, 2025

Remove TODOs per PR feedback, remove this as required CI job

be1b72c

picnixz reviewed Jun 13, 2025

View reviewed changes

Tools/ssl/multissltests.py Outdated Show resolved Hide resolved

configure.ac Show resolved Hide resolved

This was referenced Jun 16, 2025

Guard call to _hashlib in test_hashlib.py #135571

Closed

gh-135571: Guard _hashlib usage in test_hashlib.py #135572

Merged

WillChilds-Klein added 2 commits June 16, 2025 19:00

Check for SHA-512 instead of BLAKE in configure

c655484

Simplify multissltests.py builders

99df7d5

WillChilds-Klein requested a review from picnixz June 16, 2025 19:27

Modify multissltests.py comments per PR feedback

8f95caa

picnixz reviewed Jul 4, 2025

View reviewed changes

WillChilds-Klein commented Jul 7, 2025

View reviewed changes

Misc/NEWS.d/next/Tests/2025-06-11-16-52-49.gh-issue-135401.ccMXmL.rst Outdated Show resolved Hide resolved

Update Misc/NEWS.d/next/Tests/2025-06-11-16-52-49.gh-issue-135401.ccM…

4d9147c

…XmL.rst

WillChilds-Klein requested a review from picnixz July 7, 2025 00:38

picnixz reviewed Jul 7, 2025

View reviewed changes

Misc/NEWS.d/next/Tests/2025-06-11-16-52-49.gh-issue-135401.ccMXmL.rst Outdated Show resolved Hide resolved

Tools/ssl/multissltests.py Outdated Show resolved Hide resolved

WillChilds-Klein and others added 2 commits July 7, 2025 10:10

Update Misc/NEWS.d/next/Tests/2025-06-11-16-52-49.gh-issue-135401.ccM…

7311e42

…XmL.rst Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

Update Tools/ssl/multissltests.py

13dfd95

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

WillChilds-Klein requested a review from picnixz July 7, 2025 14:11

hugovk reviewed Jul 7, 2025

View reviewed changes

Tools/ssl/multissltests.py Outdated Show resolved Hide resolved

Tools/ssl/multissltests.py Outdated Show resolved Hide resolved

WillChilds-Klein and others added 2 commits July 7, 2025 11:57

Update Tools/ssl/multissltests.py

22470c1

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

Update Tools/ssl/multissltests.py

e24bde5

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>

WillChilds-Klein requested review from hugovk and zware July 8, 2025 19:34

hugovk approved these changes Jul 11, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Jul 11, 2025

Merge branch 'main' into add-awslc-ci-job

69f69e6

zware reviewed Jul 11, 2025

View reviewed changes

WillChilds-Klein and others added 3 commits July 11, 2025 16:38

Update Tools/ssl/multissltests.py

a0d7b5f

Co-authored-by: Zachary Ware <zachary.ware@gmail.com>

Update Tools/ssl/multissltests.py

a2efcda

Co-authored-by: Zachary Ware <zachary.ware@gmail.com>

Update Tools/ssl/multissltests.py

38afe99

Co-authored-by: Zachary Ware <zachary.ware@gmail.com>

zware merged commit db47f4d into python:main Jul 11, 2025
75 checks passed

bedevere-app bot removed the awaiting merge label Jul 11, 2025

This comment was marked as off-topic.

Sign in to view

	self._subprocess_call(["make", f"-j{self.jobs}"], cwd=cwd, env=env)
	self._subprocess_call(["make", "-j{}".format(self.jobs)], cwd=cwd, env=env)

Uh oh!

gh-135401: Test AWS-LC as a cryptography library in CI #135402

gh-135401: Test AWS-LC as a cryptography library in CI #135402

Conversation

WillChilds-Klein commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillChilds-Klein Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

picnixz commented Jun 13, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

WillChilds-Klein commented Jul 7, 2025

Uh oh!

picnixz commented Jul 7, 2025

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillChilds-Klein Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zware commented Jul 11, 2025

Uh oh!

Uh oh!

This comment was marked as off-topic.

hugovk commented Jul 12, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

WillChilds-Klein commented Jun 11, 2025 •

edited

Loading

WillChilds-Klein Jun 11, 2025 •

edited

Loading

WillChilds-Klein Jul 11, 2025 •

edited

Loading