gh-127971: fix off-by-one read beyond the end of a string during search #132574

duaneg · 2025-04-16T00:19:02Z

Fix cases where the string search algorithm reads one past the end of the character/byte array under certain conditions.

Issue: ASan: heap-buffer-overflow in ucs2lib_default_find #127971

Fix off-by-one read beyond the end of a string.

efimov-mikhail · 2025-06-19T11:23:24Z

Objects/stringlib/fastsearch.h

@@ -595,7 +595,7 @@ STRINGLIB(default_find)(const STRINGLIB_CHAR* s, Py_ssize_t n,
                continue;
            }
            /* miss: check if next character is part of pattern */
-            if (!STRINGLIB_BLOOM(mask, ss[i+1])) {
+            if (i < w && !STRINGLIB_BLOOM(mask, ss[i+1])) {


I'm totally not sure that else branch is appropiate here for 'i == w2.
Even since we have i > w for this m this logic isn't clear for me.
What about simple replacement i <= w for loop condition with classical i < w?
It could be enough and cleaner.

I think the else branch will be OK, since all it does is advance the index, and it is intended and expected that this could potentially advance it past the end of the string, in which case the for loop will terminate.

We can't just replace the the loop condition with i < w since the w here is the last valid index that the pattern could appear at, and needs to be checked. Otherwise we would miss valid matches, and indeed such a change breaks a large number of unit tests.

Note the conditionals that were changed are miss conditions, i.e. the algorithm has determined the character at the index cannot be part of the pattern at this location in the string. The conditionals modified are checking whether the following character could potentially be part of a pattern hit, so as to determine whether to skip it entirely by advancing the full length of the pattern or only as much as possible while still considering it as a valid potential hit. In the case where we are at the end of the buffer it doesn't actually matter which branch we take, since either way it will advance past it and terminate. We just need to avoid reading the invalid following character when it doesn't exist.

Maybe it's better to rewrite condition as i+1 <= w? It seems to be more obvious way of checking for me.
It's very similar to "for" condition, but for another argument and before direct using of i+1 as index.

... the w here is the last valid index that the pattern could appear at

IMO, rewrited condition slightly reduces cognitive load.

efimov-mikhail · 2025-06-19T11:25:05Z

I'm not at all an expert in this code, so I've tried to apply your patch by myself.
It definitely fixes related issue.

But I suggest some changes.
Moreover, I can see very similar code at adaptive_find function.
IMO, we should investigate both cases and fix them in the same PR if needed.

duaneg · 2025-06-20T00:23:32Z

I'm not at all an expert in this code, so I've tried to apply your patch by myself. It definitely fixes related issue.

I have been hesitant to ping specific developers, as I don't want to annoy people or violate etiquette, but if you think it would be appropriate there are a couple of devs who have worked on this code frequently and/or recently I could tag in?

But I suggest some changes. Moreover, I can see very similar code at adaptive_find function. IMO, we should investigate both cases and fix them in the same PR if needed.

That is an excellent find, thanks! Agreed: I'll investigate, try to get a test that reproduces the issue there also, and fix it.

efimov-mikhail · 2025-06-20T06:46:12Z

I've just used git blame and find some names.
I hope that's appropriate.
@serhiy-storchaka @sweeneyde

duaneg · 2025-06-20T11:20:03Z

I can trigger the off-by-one reads in the adaptive algorithm easily enough, but I haven't been able to get it to cause an ASAN error as yet, so no luck so far on a test for that.

The third variant, default_rfind, already includes the equivalent guard:

            /* miss: check if previous character is part of pattern */          
            if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) {                      
                i = i - m;                                                      
            }                                                                   
            else {                                                              
                i = i - skip;                                                   
            }                                                                   
        }                                                                       
        else {                                                                  
            /* skip: check if previous character is part of pattern */          
            if (i > 0 && !STRINGLIB_BLOOM(mask, s[i-1])) {                      
                i = i - m;                                                      
            }                                                                   
        }                                                                       
    }

efimov-mikhail · 2025-06-20T12:43:09Z

Does default_rfind have tests on it? If not, those worth to be added too.

IMO, we can add check and tests for adaptive_find even if there is no real crash under ASan.

duaneg · 2025-06-21T00:21:01Z

default_rfind does have tests, including ones that exercise the relevant code path. In fact any str.rfind with a non-matching pattern longer than one character (but shorter or equal to input length so it doesn't short-circuit) does so. adaptive_find also has tests, albeit none that trigger the code branches in question.

Looking closer at adaptive_find, on second thought I don't think the check is strictly required there. adaptive_find is only ever called directly from FASTSEARCH. For that function to use the adaptive_find algorithm the input length has to be > 2.5k length, and the pattern length has to be more than one character (actually 6+ or 100+ depending on input size). Given that, the character "one past the last possible position the pattern could start" will always be a valid character in the input string. As mentioned above, it doesn't actually matter which branch is taken, so it doesn't matter what its value is.

Presumably the extra clause in the conditional has some measurable performance impact, however slight, and given the input string minimum length requirements it will likely add up to much more than would be saved by avoiding one unnecessary character read. So, perhaps we shouldn't add it to adaptive_find, even though the current code seems safe only due to unchecked implementation details of when/under what circumstances it is called.

Nonetheless, I'll push out an update to this that adds an adaptive_find test, to ensure that code path is exercised, even though there is nothing we can check to verify it is doing what it is supposed to.

…asons for and limitations of the tests.

efimov-mikhail · 2025-06-21T08:38:04Z

I suggest to slightly rewrite i < w condition as i + 1 <= w for default_find and maybe i > 0 as i - 1 >= 0 for default_rfind.
Moreover, some short comment at adaptive_find code itself also seems valuable.

But those are only cosmetical changes, in other aspects LGTM.

…mment explaining why a guard is not necessary in adaptive_find.

duaneg · 2025-06-21T23:50:26Z

I suggest to slightly rewrite i < w condition as i + 1 <= w for default_find and maybe i > 0 as i - 1 >= 0 for default_rfind. Moreover, some short comment at adaptive_find code itself also seems valuable.

But those are only cosmetical changes, in other aspects LGTM.

Sure thing, will update, thanks!

efimov-mikhail

LGTM. I'd like to have just little rewording in the comment.

efimov-mikhail · 2025-07-12T12:11:50Z

Objects/stringlib/fastsearch.h

+               start in. I.e. if i == w it is safe to read ss[i + 1] since the
+               input and pattern length requirements on when this variant
+               algorithm will be called ensure it will always be a valid part
+               of the input. In that case it doesn't matter what the character


This sentence is hard for reading, IMO

I think this comment is not needed or you can summarize it as "i + 1 < len(ss) because of the input and pattern length requirements". Or add assert(i + 1 <= ...); By the way, you might want to add some asserts on n and m because it's not exactly clear that this won't be an issue.

picnixz

I am not fond of tests that may trigger a crash and the comments explaining why we have them are too long IMO. I don't see why we can't find an explicit input that triggers the code paths we want (except that it may be hard to craft such input). If it's too hard, we can just.. remove the tests. Or we should indicate in the C code that tests must be updated if we change the implementation.

picnixz · 2025-07-12T12:33:54Z

Lib/test/string_tests.py

+    def test_replacement_on_buffer_boundary(self):
+
+        # gh-127971: Check we don't read past the end of the buffer when a
+        # potential match misses on the last character. Note this will likely
+        # not cause a failure unless ASAN is enabled, and even that may be
+        # dependent on implementation details subject to change.
+        any_3_nonblank_codepoints = '!!!'
+        seven_codepoints = any_3_nonblank_codepoints + ' ' + any_3_nonblank_codepoints
+        a = (' ' * 243) + seven_codepoints + (' ' * 7)
+        b = ' ' * 6 + chr(256)


Suggested change

def test_replacement_on_buffer_boundary(self):

# gh-127971: Check we don't read past the end of the buffer when a

# potential match misses on the last character. Note this will likely

# not cause a failure unless ASAN is enabled, and even that may be

# dependent on implementation details subject to change.

any_3_nonblank_codepoints = '!!!'

seven_codepoints = any_3_nonblank_codepoints + ' ' + any_3_nonblank_codepoints

a = (' ' * 243) + seven_codepoints + (' ' * 7)

b = ' ' * 6 + chr(256)

def test_replacement_on_buffer_boundary(self):

# gh-127971: Check we don't read past the end of the buffer when a

# potential match misses on the last character. Note this will likely

# not cause a failure unless ASAN is enabled, and even that may be

# dependent on implementation details subject to change.

any_3_nonblank_codepoints = '!!!'

seven_codepoints = any_3_nonblank_codepoints + ' ' + any_3_nonblank_codepoints

a = (' ' * 243) + seven_codepoints + (' ' * 7)

b = ' ' * 6 + chr(256)

Note this will likely not cause a failure unless ASAN is enabled

In this case, make this test an ASAN test only. I would also shorten the comment.

picnixz · 2025-07-12T12:34:03Z

Lib/test/string_tests.py

+    def test_adaptive_find_on_buffer_boundary(self):
+
+        # gh-127971: This exercises the adaptive search algorithm to trigger a
+        # corner-case where it might examine the character *after* the last
+        # position that could be the start of the pattern.
+        #
+        # Unfortunately there is nothing to *test* to confirm whether the
+        # character is read or not, nor in fact does it matter for correctness
+        # with the implementation at time of writing: the adaptive algorithm is
+        # only triggered if the input is over a certain size and with a pattern
+        # with more than one character, so with the current implementation even
+        # though the final character read is not necessary or significant, it
+        # won't cause a fault.
+        #
+        # This test at least intentionally exercises this path, and might
+        # possibly catch a regression if the implementation changes and breaks
+        # those assumptions.


Suggested change

def test_adaptive_find_on_buffer_boundary(self):

# gh-127971: This exercises the adaptive search algorithm to trigger a

# corner-case where it might examine the character *after* the last

# position that could be the start of the pattern.

#

# Unfortunately there is nothing to *test* to confirm whether the

# character is read or not, nor in fact does it matter for correctness

# with the implementation at time of writing: the adaptive algorithm is

# only triggered if the input is over a certain size and with a pattern

# with more than one character, so with the current implementation even

# though the final character read is not necessary or significant, it

# won't cause a fault.

#

# This test at least intentionally exercises this path, and might

# possibly catch a regression if the implementation changes and breaks

# those assumptions.

def test_adaptive_find_on_buffer_boundary(self):

# gh-127971: This exercises the adaptive search algorithm to trigger a

# corner-case where it might examine the character *after* the last

# position that could be the start of the pattern.

#

# Unfortunately there is nothing to *test* to confirm whether the

# character is read or not, nor in fact does it matter for correctness

# with the implementation at time of writing: the adaptive algorithm is

# only triggered if the input is over a certain size and with a pattern

# with more than one character, so with the current implementation even

# though the final character read is not necessary or significant, it

# won't cause a fault.

#

# This test at least intentionally exercises this path, and might

# possibly catch a regression if the implementation changes and breaks

# those assumptions.

Why "might"? is it a corner case or not? I suggest to either find an input that triggers exactly the code paths we want, or to remove this test as it may or may not trigger the issue. Also the comment is too long IMO.

picnixz · 2025-07-12T12:34:25Z

Misc/NEWS.d/next/Core_and_Builtins/2025-04-16-12-01-13.gh-issue-127971.pMDOQ0.rst

@@ -0,0 +1 @@
+Fix off-by-one read beyond the end of a string in string search


Suggested change

Fix off-by-one read beyond the end of a string in string search

Fix off-by-one read beyond the end of a string in string search.

picnixz · 2025-07-12T12:34:52Z

Objects/stringlib/fastsearch.h

@@ -676,7 +685,9 @@ STRINGLIB(adaptive_find)(const STRINGLIB_CHAR* s, Py_ssize_t n,
            }
        }
        else {
-            /* skip: check if next character is part of pattern */
+            /* Skip: check if next character is part of pattern.
+               See comment above re safety of accessing ss[i+1] when i == w.


What does it mean "above re safety"

picnixz · 2025-07-12T12:38:04Z

Lib/test/string_tests.py

+        # though the final character read is not necessary or significant, it
+        # won't cause a fault.
+        #
+        # This test at least intentionally exercises this path, and might


which "path"?

picnixz · 2025-07-12T12:38:49Z

Objects/stringlib/fastsearch.h

+               start in. I.e. if i == w it is safe to read ss[i + 1] since the
+               input and pattern length requirements on when this variant
+               algorithm will be called ensure it will always be a valid part
+               of the input. In that case it doesn't matter what the character


I think this comment is not needed or you can summarize it as "i + 1 < len(ss) because of the input and pattern length requirements". Or add assert(i + 1 <= ...); By the way, you might want to add some asserts on n and m because it's not exactly clear that this won't be an issue.

duaneg added 3 commits April 15, 2025 20:56

pythongh-127971: do not read past the end of a string

f0d20b2

Fix off-by-one read beyond the end of a string.

Add test case

462dddf

Add blurb

1d0210a

bedevere-app bot added the awaiting review label Apr 16, 2025

bedevere-app bot mentioned this pull request Apr 16, 2025

ASan: heap-buffer-overflow in ucs2lib_default_find #127971

Open

duaneg mentioned this pull request Jun 18, 2025

gh-135335: flush stdout/stderr in forkserver after preloading modules #135338

Merged

efimov-mikhail reviewed Jun 19, 2025

View reviewed changes

serhiy-storchaka self-requested a review June 20, 2025 07:10

Add test case for adaptive_find and comments better explaining the re…

212452d

…asons for and limitations of the tests.

Tweak conditional phrasing to match loop terminating criteria, add co…

c07c23e

…mment explaining why a guard is not necessary in adaptive_find.

efimov-mikhail approved these changes Jul 12, 2025

View reviewed changes

bedevere-app bot added awaiting core review and removed awaiting review labels Jul 12, 2025

picnixz reviewed Jul 12, 2025

View reviewed changes

		@@ -0,0 +1 @@
		Fix off-by-one read beyond the end of a string in string search

Uh oh!

gh-127971: fix off-by-one read beyond the end of a string during search #132574

Are you sure you want to change the base?

gh-127971: fix off-by-one read beyond the end of a string during search #132574

Conversation

duaneg commented Apr 16, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efimov-mikhail Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efimov-mikhail commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

duaneg commented Jun 20, 2025

Uh oh!

efimov-mikhail commented Jun 20, 2025

Uh oh!

duaneg commented Jun 20, 2025

Uh oh!

efimov-mikhail commented Jun 20, 2025

Uh oh!

duaneg commented Jun 21, 2025

Uh oh!

efimov-mikhail commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

duaneg commented Jun 21, 2025

Uh oh!

efimov-mikhail left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

picnixz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

duaneg commented Apr 16, 2025 •

edited by bedevere-app bot

Loading

efimov-mikhail Jun 21, 2025 •

edited

Loading

efimov-mikhail commented Jun 19, 2025 •

edited

Loading

efimov-mikhail commented Jun 21, 2025 •

edited

Loading

efimov-mikhail left a comment •

edited

Loading