Skip to content

gh-55531: Implement normalize_encoding in C #136643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

gh-55531: Implement normalize_encoding in C #136643

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
92873d6
C
StanFromIreland Jul 14, 2025
4bae23a
Correct clinic note
StanFromIreland Jul 14, 2025
b5f3df3
Little fixes
StanFromIreland Jul 14, 2025
2ad72b2
Keep the messiness
StanFromIreland Jul 14, 2025
3660160
Clean up tests
StanFromIreland Jul 14, 2025
4e12b9e
Remove unnecessary message
StanFromIreland Jul 14, 2025
1c9e55a
Review
StanFromIreland Jul 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Review
  • Loading branch information
@StanFromIreland
StanFromIreland committed Jul 15, 2025
commit 1c9e55ab8ffafd2bb0e68c688fadab90399cfc16
5 changes: 2 additions & 3 deletions Lib/encodings/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,19 +26,18 @@

(c) Copyright CNRI, All Rights Reserved. NO WARRANTY.

"""
"""#"

import codecs
from _codecs import _normalize_encoding
import sys
from _codecs import _normalize_encoding
from . import aliases

_cache = {}
_unknown = '--unknown--'
_import_tail = ['*']
_aliases = aliases.aliases


class CodecRegistryError(LookupError, SystemError):
pass

Expand Down
1 change: 0 additions & 1 deletion Lib/test/test_codecs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3900,7 +3900,6 @@ def test_encodings_normalize_encoding(self):
self.assertEqual(normalize('utf_8'), 'utf_8')
self.assertEqual(normalize('utf\xE9\u20AC\U0010ffff-8'), 'utf_8')
self.assertEqual(normalize('utf 8'), 'utf_8')

# encodings.normalize_encoding() doesn't convert
# characters to lower case.
self.assertEqual(normalize('UTF 8'), 'UTF_8')
Expand Down
19 changes: 15 additions & 4 deletions Modules/_codecsmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1037,30 +1037,41 @@ static PyObject *
_codecs__normalize_encoding_impl(PyObject *module, PyObject *encoding)
/*[clinic end generated code: output=d27465d81e361f8e input=3ff3f4d64995b988]*/
{
const char *cstr = PyUnicode_AsUTF8(encoding);
Py_ssize_t len;
const char *cstr = PyUnicode_AsUTF8AndSize(encoding, &len);
if (cstr == NULL) {
return NULL;
}

size_t len = strlen(cstr);
if (len > PY_SSIZE_T_MAX) {
PyErr_SetString(PyExc_OverflowError, "encoding is too large");
return NULL;
}

PyUnicodeWriter *writer = PyUnicodeWriter_Create(len + 1);
if (writer == NULL) {
return NULL;
}

char *normalized = PyMem_Malloc(len + 1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be possible to use a VLA for this, but I'm not sure it's worth the additional complexity. @picnixz WDYT?

if (normalized == NULL) {
PyUnicodeWriter_Discard(writer);
return PyErr_NoMemory();
}

if (!_Py_normalize_encoding(cstr, normalized, len + 1, 0)) {
PyMem_Free(normalized);
PyUnicodeWriter_Discard(writer);
return NULL;
}

PyObject *v = PyUnicode_FromString(normalized);
if (PyUnicodeWriter_WriteUTF8(writer, normalized, (Py_ssize_t)strlen(normalized)) < 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The size shouldn't need to be recalculated here. It's always len + 1, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cpython/Lib/test/test_codecs.py

Lines 3901 to 3902 in 28937d3

self.assertEqual(normalize('utf\xE9\u20AC\U0010ffff-8'), 'utf_8')
self.assertEqual(normalize('utf 8'), 'utf_8')

No, it must be done to match the current behavior, where it can change.

PyUnicodeWriter_Discard(writer);
PyMem_Free(normalized);
return NULL;
}
PyMem_Free(normalized);
return v;
return PyUnicodeWriter_Finish(writer);
}

/* --- Module API --------------------------------------------------------- */
Expand Down
Loading
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy