gh-131738: optimize builtin any/all/tuple calls with a generator expression arg by iritkatriel · Pull Request #131737 · python/cpython

iritkatriel · 2025-03-25T20:35:17Z

This PR implements the optimization described in faster-cpython/ideas#545.

Issue: optimize any/all/tuple calls with a generator expression arg #131738

Misc/NEWS.d/next/Core_and_Builtins/2025-03-25-20-38-06.gh-issue-131738.eCb0OQ.rst

TeamSpen210 · 2025-03-25T20:57:09Z

I don't think LOAD_COMMON_CONST is safe right now, since users could have messed with the builtins module. Probably better to stash the funcs in interp->callable_cache like other builtins used for optimisation?

brandtbucher

Very cool! Any plans to do this for set or list?

dict/max/min/sum are trickier, but may be worth exploring too.

Lib/opcode.py

Python/bytecodes.c

Python/codegen.c

bedevere-app · 2025-03-25T22:17:46Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

brandtbucher · 2025-03-26T00:09:37Z

This should fix the JIT builds:

diff --git a/Tools/jit/_targets.py b/Tools/jit/_targets.py
index aa2b56abf44..73a4210eee0 100644
--- a/Tools/jit/_targets.py
+++ b/Tools/jit/_targets.py
@@ -522,7 +522,14 @@ def get_target(host: str) -> _COFF | _ELF | _MachO:
         args = ["-fms-runtime-lib=dll"]
         target = _COFF(host, args=args)
     elif re.fullmatch(r"x86_64-.*-linux-gnu", host):
-        args = ["-fno-pic", "-mcmodel=medium", "-mlarge-data-threshold=0"]
+        args = [
+            # Jump tables generate R_X86_64_32S relocations, which assume a
+            # signed 32-bit address space:
+            "-fno-jump-tables",
+            "-fno-pic",
+            "-mcmodel=medium",
+            "-mlarge-data-threshold=0",
+        ]
         target = _ELF(host, args=args)
     else:
         raise ValueError(host)

Python/bytecodes.c

markshannon · 2025-03-26T11:07:44Z

The overall approach looks good.

Do you have stats for this PR?
If there is enough of an increase in the execution counts for CALL_INTRINSIC_1 then we should move CALL_INTRINSIC_1 INTRINSIC_LIST_TO_TUPLE into a new LIST_TO_TUPLE instruction.

rhettinger · 2025-03-26T20:45:17Z

dict/max/min/sum are trickier, but may be worth exploring too.

The latter three would be nice wins.

Python/bytecodes.c

iritkatriel · 2025-03-27T11:07:05Z

Perf results are here: https://github.com/faster-cpython/benchmarking-public/blob/main/results/bm-20250326-3.14.0a6%2B-50dd66b/README.md

iritkatriel · 2025-03-27T11:10:17Z

Do you have stats for this PR? If there is enough of an increase in the execution counts for CALL_INTRINSIC_1 then we should move CALL_INTRINSIC_1 INTRINSIC_LIST_TO_TUPLE into a new LIST_TO_TUPLE instruction.

iritkatriel · 2025-03-27T11:10:43Z

I have made the requested changes; please review again.

bedevere-app · 2025-03-27T11:10:48Z

Thanks for making the requested changes!

@brandtbucher: please review the changes made to this pull request.

brandtbucher

Looks good. You can revert the JIT change, it's not needed anymore.

(Also I'm running JIT benchmarks just because I'm curious, but no need to block on those results at all.)

brandtbucher · 2025-03-27T19:56:13Z

Objects/genobject.c

 #include "pycore_warnings.h"      // _PyErr_WarnUnawaitedCoroutine()


+#include "opcode_ids.h"           // RESUME, etc


Is this needed?

Yes, I had to remove this include from another header so it needs to be included when needed.

Tools/jit/_targets.py

brandtbucher · 2025-03-27T20:59:38Z

(Also, I think you need to make regen-cases.)

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>

…r expression arg (python#131737)

iritkatriel added 2 commits March 25, 2025 20:28

add CONSTANT_BUILTIN_ALL/ANY/TUPLE oparg for LOAD_COMMON_CONST

8de7276

implement all/any/tuple optimization

df6bccf

iritkatriel requested a review from markshannon as a code owner March 25, 2025 20:35

bedevere-app bot added the awaiting core review label Mar 25, 2025

iritkatriel changed the title ~~optimize any/all/tuple calls with a generator expression arg~~ gh-131738: optimize builtin any/all/tuple calls with a generator expression arg Mar 25, 2025

bedevere-app bot mentioned this pull request Mar 25, 2025

optimize any/all/tuple calls with a generator expression arg #131738

Closed

📜🤖 Added by blurb_it.

045cdc9

iritkatriel commented Mar 25, 2025

View reviewed changes

Misc/NEWS.d/next/Core_and_Builtins/2025-03-25-20-38-06.gh-issue-131738.eCb0OQ.rst Outdated Show resolved Hide resolved

iritkatriel added 3 commits March 25, 2025 20:50

add comment

d142f19

fix warning

cb4eddb

fixup

c3bb232

use interp->callable_cache

a217fc4

iritkatriel requested a review from ericsnowcurrently as a code owner March 25, 2025 22:15

brandtbucher requested changes Mar 25, 2025

View reviewed changes

bedevere-app bot added awaiting changes and removed awaiting core review labels Mar 25, 2025

tweaks from review

9f71f16

markshannon reviewed Mar 26, 2025

View reviewed changes

Python/bytecodes.c Outdated Show resolved Hide resolved

iritkatriel added 3 commits March 26, 2025 17:47

avoid duplicating name lookup

10ef702

remove pop

222ef3b

fix build

ef78fd1

iritkatriel requested a review from savannahostrowski as a code owner March 26, 2025 18:05

use table for common consts

e4b3ccd

iritkatriel added 2 commits March 26, 2025 21:39

typo

7ae52ab

Merge remote-tracking branch 'upstream/main' into tuple

a495ef4

iritkatriel commented Mar 26, 2025

View reviewed changes

Python/bytecodes.c Outdated Show resolved Hide resolved

Update Python/bytecodes.c

50dd66b

bedevere-app bot added awaiting change review and removed awaiting changes labels Mar 27, 2025

bedevere-app bot requested a review from brandtbucher March 27, 2025 11:10

brandtbucher approved these changes Mar 27, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting change review labels Mar 27, 2025

iritkatriel and others added 2 commits March 28, 2025 10:07

Update Tools/jit/_targets.py

d5071b3

Co-authored-by: Brandt Bucher <brandtbucher@gmail.com>

regen-cases

44b9e8d

iritkatriel enabled auto-merge (squash) March 28, 2025 10:13

iritkatriel merged commit 2c8f329 into python:main Mar 28, 2025
64 checks passed

bedevere-app bot removed the awaiting merge label Mar 28, 2025

davfsa mentioned this pull request Mar 31, 2025

SIM110: Recommend proper usage of all and any astral-sh/ruff#16904

Open

diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 1, 2025

pythongh-131738: optimize builtin any/all/tuple calls with a generato…

8a283af

…r expression arg (python#131737)

opk12 mentioned this pull request Jul 12, 2025

New Rule: Prefer list comprehension over generator comprehensions to create tuples astral-sh/ruff#11839

Open

		#include "pycore_warnings.h" // _PyErr_WarnUnawaitedCoroutine()


		#include "opcode_ids.h" // RESUME, etc

Uh oh!

Conversation

iritkatriel commented Mar 25, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

TeamSpen210 commented Mar 25, 2025

Uh oh!

brandtbucher left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-app bot commented Mar 25, 2025

Uh oh!

brandtbucher commented Mar 26, 2025

Uh oh!

Uh oh!

markshannon commented Mar 26, 2025

Uh oh!

rhettinger commented Mar 26, 2025

Uh oh!

Uh oh!

iritkatriel commented Mar 27, 2025

Uh oh!

iritkatriel commented Mar 27, 2025

Uh oh!

iritkatriel commented Mar 27, 2025

Uh oh!

bedevere-app bot commented Mar 27, 2025

Uh oh!

brandtbucher left a comment

Choose a reason for hiding this comment

Uh oh!

brandtbucher Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

iritkatriel Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

brandtbucher commented Mar 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

iritkatriel commented Mar 25, 2025 •

edited by bedevere-app bot

Loading

brandtbucher left a comment •

edited

Loading