Do not allocate non-PyObjects with PyObject_Malloc()

[colesbury]

Feature or enhancement

The free-threaded build requires that Python objects -- and only Python objects -- be allocated through the Python object allocation APIs (like PyObject_Malloc() or PyType_GenericNew()) ¹.

There are a few places internally ² that use PyObject_Malloc() for non Python objects. We should switch those call sites to use PyMem_Malloc()/Free() instead.

Note that there is not a significant difference between using PyObject_Malloc() and PyMem_Malloc() in the default build. Both calls use obmalloc under the hood, so switching from one to the other should not matter for the default build.

Here are some examples, but this list may not be exhaustive:

cpython/Modules/_sre/sre_lib.h

Line 1125 in 841eacd

ctx->u.rep = (SRE_REPEAT*) PyObject_Malloc(sizeof(*ctx->u.rep));

cpython/Modules/_elementtree.c

Line 270 in 841eacd

self->extra = PyObject_Malloc(sizeof(ElementObjectExtra));

cpython/Modules/_elementtree.c

Lines 498 to 499 in 841eacd

	children = PyObject_Realloc(self->extra->children,
	size * sizeof(PyObject*));

https://github.com/python/cpython/blob/main/Modules/mathmodule.c#L2573

cpython/Modules/pyexpat.c

Lines 23 to 24 in 841eacd

	static XML_Memory_Handling_Suite ExpatMemoryHandler = {
	PyObject_Malloc, PyObject_Realloc, PyObject_Free};

cpython/Objects/bytearrayobject.c

Line 135 in 841eacd

new->ob_bytes = PyObject_Malloc(alloc);

cpython/Parser/lexer/lexer.c

Line 132 in 841eacd

char *result = (char *)PyObject_Malloc((input_length + 1) * sizeof(char));

Linked PRs

• gh-114569: Use PyMem_* APIs for non-PyObjects in Modules/, Objects/ and Parser/ #114574
• gh-114569: Use PyMem_* APIs for non-PyObjects in compiler #114587
• gh-114569: Use PyMem_* APIs for non-PyObjects in unicodeobject.c #114690

Footnotes

1. See https://peps.python.org/pep-0703/#backwards-compatibility:~:text=Non%2DPython%20objects%20must%20not%20be%20allocated%20through%20those%20APIs ↩

2. @DinoV fixed the use in dictobject.c in https://github.com/python/cpython/pull/114543 ↩

[erlend-aasland]

There is some strange usage in unicodeobject.c as well; I did not look closely at it yet. Perhaps it is ok, perhaps not.

[colesbury]

Yeah, it looks like a two of those should use PyMem as well, but not the struct encoding_map one -- that's a Python object.

cpython/Objects/unicodeobject.c

Line 5206 in 30b7b4f

char *cache = PyObject_Malloc(len + 1);

cpython/Objects/unicodeobject.c

Line 14677 in 30b7b4f

data = PyObject_Malloc((length + 1) * char_size);

[erlend-aasland]

Did we catch all of them?

[colesbury]

Yeah, looks like it!