iter `reduce` can segfault if accessing `builtins.dict['iter']` mutates the iter object #101765

ionite34 · 2023-02-09T22:04:14Z

Crash report

corrupt = iter(lambda:0, 0)

class Cstr:
    def __hash__(self):
        return hash('iter')
    def __eq__(self, other):
        [*corrupt]
        return other == 'iter'

builtins = __builtins__.__dict__ if hasattr(__builtins__, '__dict__') else __builtins__
oiter = builtins['iter']
del builtins['iter']
builtins[Cstr()] = oiter

print(corrupt.__reduce__())

Expected result

This should return a valid __reduce__ tuple of the exhausted iterator. Instead behavior is inconsistent between segmentation faults, SystemErrors, and sometimes returning the iterator without being exhausted.

Error messages

3.11, windows, PYTHONMALLOC=debug
3.12.0a4, windows, PYTHONMALLOC=debug

Windows fatal exception: access violation
> exit code -1073741819 (0xC0000005)

3.12.04a4, windows, compiled with debug mode

    print(corrupt.__reduce__())
          ^^^^^^^^^^^^^^^^^^^^
SystemError: NULL object passed to Py_BuildValue

3.11, ubuntu

(<built-in function iter>, (<function  at 0x7fb772c3c4a0>, 0))
> terminated by signal SIGSEGV (Address boundary error)

3.12.0a4, ubuntu

(<built-in function iter>, (<function  at 0x7f3480d71f80>, 0))

3.12.0a4, ubuntu, PYTHONMALLOC=debug

Fatal Python error: Segmentation fault

Linked PRs

The text was updated successfully, but these errors were encountered:

godlygeek · 2023-02-09T22:08:19Z

This (horrific) reproducer provokes undefined defined behavior from this statement:

cpython/Objects/iterobject.c

Lines 243 to 244 in f1f3af7

    
           return Py_BuildValue("N(OO)", _PyEval_GetBuiltin(&_Py_ID(iter)), 
        
                                it->it_callable, it->it_sentinel);

The call to _PyEval_GetBuiltin to find the iter builtin is calling Cstr.__eq__, which exhausts the iterator, causing the Py_CLEAR in calliter_iternext to be executed, setting it->it_callable and it->it_sentinel to NULL. But the order of evaluation of arguments in a function call isn't specified, and modifying an argument by evaluating another argument is a bug.

On some platforms, it->it_callable and it->it_sentinel are being evaluated before _PyEval_GetBuiltin, and so Py_BuildValue is being passed pointers to objects that got freed inside the _PyEval_GetBuiltin call.

…when internal access of `builtins.__dict__` exhausts the iterator (#101769)

…`__reduce__` when internal access of `builtins.__dict__` exhausts the iterator (pythonGH-101769). (cherry picked from commit 54dfa14) Co-authored-by: Ionite <dev@ionite.io>

…uce__` when internal access of `builtins.__dict__` exhausts the iterator (GH-101769) (#102228) (cherry picked from commit 54dfa14)

…uce__` when internal access of `builtins.__dict__` exhausts the iterator (GH-101769) (#102229) (cherry picked from commit 54dfa14)

Followup from python#101769.

Followup from #101769.

…ing (pythonGH-102265) Followup from pythonGH-101769.. (cherry picked from commit d71edbd) Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

…H-102265) (#102269) (cherry picked from commit d71edbd)

…H-102265) (#102268) (cherry picked from commit d71edbd)

ionite34 added the type-crash A hard crash of the interpreter, possibly with a core dump label Feb 9, 2023

bedevere-bot mentioned this issue Feb 10, 2023

gh-101765: Fix SystemError / segmentation fault in iter __reduce__ when internal access of builtins.__dict__ exhausts the iterator #101769

Merged

7 tasks

ionite34 changed the title ~~iter.__reduce__ can segfault if accessing __builtins__.__dict__['iter'] exhausts the iter object~~ iter __reduce__ can segfault if accessing __builtins__.__dict__['iter'] mutates the iter object Feb 10, 2023

arhadthedev added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Feb 10, 2023

ionite34 added a commit to ionite34/cpython that referenced this issue Feb 10, 2023

Add iter reduce tests for issue python#101765

c5abb14

JelleZijlstra pushed a commit that referenced this issue Feb 24, 2023

gh-101765: Fix SystemError / segmentation fault in iter __reduce__ …

54dfa14

…when internal access of `builtins.__dict__` exhausts the iterator (#101769)

bedevere-bot mentioned this issue Feb 25, 2023

[3.11] gh-101765: Fix SystemError / segmentation fault in iter __reduce__ when internal access of builtins.__dict__ exhausts the iterator (GH-101769) #102228

Merged

bedevere-bot mentioned this issue Feb 25, 2023

[3.10] gh-101765: Fix SystemError / segmentation fault in iter __reduce__ when internal access of builtins.__dict__ exhausts the iterator (GH-101769) #102229

Merged

JelleZijlstra pushed a commit that referenced this issue Feb 25, 2023

[3.11] gh-101765: Fix SystemError / segmentation fault in iter `__red…

5d46122

…uce__` when internal access of `builtins.__dict__` exhausts the iterator (GH-101769) (#102228) (cherry picked from commit 54dfa14)

JelleZijlstra pushed a commit that referenced this issue Feb 25, 2023

[3.10] gh-101765: Fix SystemError / segmentation fault in iter `__red…

9f472f8

…uce__` when internal access of `builtins.__dict__` exhausts the iterator (GH-101769) (#102229) (cherry picked from commit 54dfa14)

bedevere-bot mentioned this issue Feb 25, 2023

gh-101765: Fix refcount issues in list and unicode pickling #102265

Merged

JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this issue Feb 25, 2023

pythongh-101765: Fix refcount issues in list and unicode pickling

b8ad59e

Followup from python#101769.

JelleZijlstra added a commit that referenced this issue Feb 26, 2023

gh-101765: Fix refcount issues in list and unicode pickling (#102265)

d71edbd

Followup from #101769.

bedevere-bot mentioned this issue Feb 26, 2023

[3.11] gh-101765: Fix refcount issues in list and unicode pickling (G… #102268

Merged

bedevere-bot mentioned this issue Feb 26, 2023

[3.10] gh-101765: Fix refcount issues in list and unicode pickling (G… #102269

Merged

JelleZijlstra added a commit that referenced this issue Feb 26, 2023

[3.10] gh-101765: Fix refcount issues in list and unicode pickling (G…

6fa6c2a

…H-102265) (#102269) (cherry picked from commit d71edbd)

JelleZijlstra added a commit that referenced this issue Feb 26, 2023

[3.11] gh-101765: Fix refcount issues in list and unicode pickling (G…

b36c498

…H-102265) (#102268) (cherry picked from commit d71edbd)

Jan	FEB	Mar
	26
2022	2023	2024

iter `reduce` can segfault if accessing `builtins.dict['iter']` mutates the iter object #101765

iter `reduce` can segfault if accessing `builtins.dict['iter']` mutates the iter object #101765

ionite34 commented Feb 9, 2023 •

edited by bedevere-bot

godlygeek commented Feb 9, 2023 •

edited

iter __reduce__ can segfault if accessing __builtins__.__dict__['iter'] mutates the iter object #101765

iter __reduce__ can segfault if accessing __builtins__.__dict__['iter'] mutates the iter object #101765

Comments

ionite34 commented Feb 9, 2023 • edited by bedevere-bot

Crash report

Expected result

Error messages

Linked PRs

godlygeek commented Feb 9, 2023 • edited

iter `reduce` can segfault if accessing `builtins.dict['iter']` mutates the iter object #101765

iter `reduce` can segfault if accessing `builtins.dict['iter']` mutates the iter object #101765

ionite34 commented Feb 9, 2023 •

edited by bedevere-bot

godlygeek commented Feb 9, 2023 •

edited