Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Found Heap-use-after-free errors and SEGV in Python #103824

Closed
JohenanLi opened this issue Apr 25, 2023 · 9 comments
Closed

Found Heap-use-after-free errors and SEGV in Python #103824

JohenanLi opened this issue Apr 25, 2023 · 9 comments
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@JohenanLi
Copy link

JohenanLi commented Apr 25, 2023 •
edited by bedevere-bot
Loading

Your environment

  • CPython versions tested on: 3.12.0 alpha 7
  • Operating system and architecture: ubuntu20.04.1,x86_64
  • Compiler flags: clang with ASAN and UBSAN instrument

Bug description

The AddressSanitizer (ASAN) tool has detected multiple heap-use-after-free errors and a segmentation fault (SEGV) in the Python interpreter. The heap-use-after-free errors occurred in the ascii_decode and unicode_decode_utf8 functions in the unicodeobject.c file, and the SEGV occurred in the tok_backup function in the tokenizer.c file. Additionally, a memory leak was detected in the pystate.c file.

Steps to reproduce

  1. Compile Python with ASAN enabled: ./configure && make
  2. Run Python with ASAN enabled: ./python < poc_file
  3. The heap-use-after-free errors and SEGV should be detected and logged by ASAN.

Expected behavior

No heap-use-after-free errors or SEGV should occur.

Actual behavior

ASAN detected multiple heap-use-after-free errors and a SEGV, as well as a memory leak.

Relevant logs and/or screenshots

The ASAN summary output is as follows:

AddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4474:28 in ascii_decode
AddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4506:28 in ascii_decode
AddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4483:32 in ascii_decode
AddressSanitizer: SEGV /src/cpython/Parser/tokenizer.c:1234:33 in tok_backup
AddressSanitizer: heap-use-after-free /src/cpython/Objects/unicodeobject.c:4526:37 in unicode_decode_utf8
AddressSanitizer: 3824 byte(s) leaked in 4 allocation(s).
AddressSanitizer: heap-use-after-free /src/cpython/Python/pystate.c:229:23 in bind_tstate
The full ASAN log can be found in the asan.log file.

asan.log
python_bug_poc.zip

Linked PRs
@JohenanLi JohenanLi added the type-bug An unexpected behavior, bug, or error label Apr 25, 2023
@sunmy2019
Copy link
Member

CC: @lysnikolaou

@arhadthedev arhadthedev added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump and removed type-bug An unexpected behavior, bug, or error labels Apr 25, 2023
@sobolevn
Copy link
Member

sobolevn commented Apr 25, 2023 •
edited
Loading

Sorry, I might be missing something, but your archive does not have poc_file.
Снимок экрана 2023-04-25 в 16 50 39

There are several bugN files. Do you mean to actually run them?

@JohenanLi
Copy link
Author

Sorry, I might be missing something, but your archive does not have poc_file. Снимок экрана 2023-04-25 в 16 50 39

There are several bugN files. Do you mean to actually run them?

yes，run them. python < bugN.

@chgnrdv
Copy link
Contributor

chgnrdv commented Apr 29, 2023

I can reproduce use-after-free errors detected in unicodeobject.c with bug_2, bug_6, bug_7 and bug_9 files. I don't get a segfault in tok_backup func with bug_7 on my machine, bug I get the same use-after-free.
I'll submit a PR with possible fix for these errors, if nobody minds.

Unfortunately, I don't get use-after-free error with bug_4 on my machine. I can't reproduce use-after-free in bind_tstate with bug_15 as well.

@chgnrdv chgnrdv mentioned this issue Apr 29, 2023
Merged
@JohenanLi
Copy link
Author

I can reproduce use-after-free errors detected in unicodeobject.c with bug_2, bug_6, bug_7 and bug_9 files. I don't get a segfault in tok_backup func with bug_7 on my machine, bug I get the same use-after-free. I'll submit a PR with possible fix for these errors, if nobody minds.

Unfortunately, I don't get use-after-free error with bug_4 on my machine. I can't reproduce use-after-free in bind_tstate with bug_15 as well.

Thanks a lot.

pablogsal pushed a commit that referenced this issue May 1, 2023
@chgnrdv
gh-103824: fix use-after-free error in Parser/tokenizer.c (#103993)
d5a9707
carljm added a commit to carljm/cpython that referenced this issue May 1, 2023
@carljm
Merge branch 'main' into inlinecomp2
1274e2b 
* main: (463 commits)
  pythongh-104057: Fix direct invocation of test_super (python#104064)
  pythongh-87092: Expose assembler to unit tests (python#103988)
  pythongh-97696: asyncio eager tasks factory (python#102853)
  pythongh-84436: Immortalize in _PyStructSequence_InitBuiltinWithFlags() (pythongh-104054)
  pythongh-104057: Fix direct invocation of test_module (pythonGH-104059)
  pythongh-100458: Clarify Enum.__format__() change of mixed-in types in the whatsnew/3.11.rst (pythonGH-100387)
  pythongh-104018: disallow "z" format specifier in %-format of byte strings (pythonGH-104033)
  pythongh-104016: Fixed off by 1 error in f string tokenizer (python#104047)
  pythonGH-103629: Update Unpack's repr in compliance with PEP 692 (python#104048)
  pythongh-102799: replace sys.exc_info by sys.exception in inspect and traceback modules (python#104032)
  Fix typo in "expected" word in few source files (python#104034)
  pythongh-103824: fix use-after-free error in Parser/tokenizer.c (python#103993)
  pythongh-104035: Do not ignore user-defined `__{get,set}state__` in slotted frozen dataclasses (python#104041)
  pythongh-104028: Reduce object creation while calling callback function from gc (pythongh-104030)
  pythongh-104036: Fix direct invocation of test_typing (python#104037)
  pythongh-102213: Optimize the performance of `__getattr__` (pythonGH-103761)
  pythongh-103895: Improve how invalid `Exception.__notes__` are displayed (python#103897)
  Adjust expression from `==` to `!=` in alignment with the meaning of the paragraph. (pythonGH-104021)
  pythongh-88496: Fix IDLE test hang on macOS (python#104025)
  Improve int test coverage (python#104024)
  ...
@lysnikolaou
Copy link
Contributor

Resolved in #103993.

@chgnrdv chgnrdv mentioned this issue May 23, 2023
Closed
@ajakk
Copy link

ajakk commented Jun 9, 2023

Did this ever affect other release lines?

@schribl
Copy link
Contributor

schribl commented Jul 10, 2023

Did this ever affect other release lines?

We would also be interested in the answer to this question, if possible. Thanks a lot!

@lysnikolaou
Copy link
Contributor

I just did another check on 3.11 and everything appears to be okay. As far as I can see, this only ever affected 3.12, since it was introduced in the implementation of PEP 701.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@sobolevn @arhadthedev @schribl @JohenanLi @lysnikolaou @ajakk @chgnrdv @sunmy2019