X Tutup
Skip to content

gh-126505: Fix bugs in compiling case-insensitive character classes#126557

Merged
serhiy-storchaka merged 1 commit intopython:mainfrom
serhiy-storchaka:re-charset-nonbmp
Nov 11, 2024
Merged

gh-126505: Fix bugs in compiling case-insensitive character classes#126557
serhiy-storchaka merged 1 commit intopython:mainfrom
serhiy-storchaka:re-charset-nonbmp

Conversation

@serhiy-storchaka
Copy link
Member

@serhiy-storchaka serhiy-storchaka commented Nov 7, 2024 •
edited by bedevere-app bot
Loading

  • upper-case non-BMP character was ignored
  • the ASCII flag was ignored when matching a character range whose upper bound is beyond the BMP region

@serhiy-storchaka
pythongh-126505: Fix bugs in compiling case-insensitive character cla…
e0fcd2f 
…sses

* upper-case non-BMP character was ignored
* the ASCII flag was ignored when matching a character range whose
  upper bound is beyond the BMP region
@serhiy-storchaka serhiy-storchaka added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Nov 7, 2024
@bedevere-app bedevere-app bot mentioned this pull request Nov 7, 2024
Closed
@serhiy-storchaka serhiy-storchaka merged commit 819830f into python:main Nov 11, 2024
@miss-islington-app
Copy link

miss-islington-app bot commented Nov 11, 2024

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

@serhiy-storchaka serhiy-storchaka deleted the re-charset-nonbmp branch November 11, 2024 16:27
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 11, 2024
@serhiy-storchaka @miss-islington
pythongh-126505: Fix bugs in compiling case-insensitive character cla…
47897fe 
…sses (pythonGH-126557)

* upper-case non-BMP character was ignored
* the ASCII flag was ignored when matching a character range whose
  upper bound is beyond the BMP region
(cherry picked from commit 819830f)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-app
Copy link

bedevere-app bot commented Nov 11, 2024

GH-126689 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Nov 11, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 11, 2024
@serhiy-storchaka @miss-islington
pythongh-126505: Fix bugs in compiling case-insensitive character cla…
a6006ca 
…sses (pythonGH-126557)

* upper-case non-BMP character was ignored
* the ASCII flag was ignored when matching a character range whose
  upper bound is beyond the BMP region
(cherry picked from commit 819830f)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-app
Copy link

bedevere-app bot commented Nov 11, 2024

GH-126690 is a backport of this pull request to the 3.12 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.12 only security fixes label Nov 11, 2024
serhiy-storchaka added a commit that referenced this pull request Nov 11, 2024
@miss-islington @serhiy-storchaka
[3.12] gh-126505: Fix bugs in compiling case-insensitive character cl…
7d091d5 
…asses (GH-126557) (GH-126690)

* upper-case non-BMP character was ignored
* the ASCII flag was ignored when matching a character range whose
  upper bound is beyond the BMP region
(cherry picked from commit 819830f)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this pull request Nov 11, 2024
@miss-islington @serhiy-storchaka
[3.13] gh-126505: Fix bugs in compiling case-insensitive character cl…
7db6d42 
…asses (GH-126557) (GH-126689)

* upper-case non-BMP character was ignored
* the ASCII flag was ignored when matching a character range whose
  upper bound is beyond the BMP region
(cherry picked from commit 819830f)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@wjssz
Copy link

wjssz commented Nov 15, 2024 •
edited
Loading 

    if fixup: # IGNORECASE and not LOCALE
        av = fixup(av)
        charmap[av] = 1

The charmap index, sometimes is fixup(av), sometimes is fixup(fixup(av)). Perhaps the result is the same, but it's not a good style.

You can fixup(av) in below, so that the code above doesn't need to be modified:

			if op is RANGE:
				if fixes: # not ASCII
					op = RANGE_UNI_IGNORE
				hascased = True
			else:
				assert op is LITERAL
				if not hascased and iscased(av):
					hascased = True
				av = fixup(av)  # <- fixup() here
		tail.append((op, av))
	break

@serhiy-storchaka
Copy link
Member Author

serhiy-storchaka commented Nov 15, 2024

fixup() is only called twice when we get a non-BMP character when len(charmap) == 256. Are you talking about this situation? Than it occurs only once, and since fixup(fixup(av)) is equal to fixup(av), it does not matter. In contrary, if write the code in a way you proposed (this is how I wrote it initially too), fixup(av) will be called twice for every non-BMP literal. The difference may be not large, but this is clearly less efficient.

@wjssz
Copy link

wjssz commented Nov 16, 2024

Ok, this is not an important issue.

picnixz pushed a commit to picnixz/cpython that referenced this pull request Dec 8, 2024
@serhiy-storchaka @picnixz
pythongh-126505: Fix bugs in compiling case-insensitive character cla…
b6c7058 
…sses (pythonGH-126557)

* upper-case non-BMP character was ignored
* the ASCII flag was ignored when matching a character range whose
  upper bound is beyond the BMP region
ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025
@serhiy-storchaka @ebonnal
pythongh-126505: Fix bugs in compiling case-insensitive character cla…
e131426 
…sses (pythonGH-126557)

* upper-case non-BMP character was ignored
* the ASCII flag was ignored when matching a character range whose
  upper bound is beyond the BMP region
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@serhiy-storchaka @wjssz
X Tutup