3.12.0a2.rst

The IDNA codec decoder used on DNS hostnames by :mod:`socket` or :mod:`asyncio` related name resolution functions no longer involves a quadratic algorithm. This prevents a potential CPU denial of service if an out-of-spec excessive length hostname involving bidirectional characters were decoded. Some protocols such as :mod:`urllib` http :samp:`3{xx}` redirects potentially allow for an attacker to supply such a name.

Individual labels within an IDNA encoded DNS name will now raise an error early during IDNA decoding if they are longer than 1024 unicode characters given that each decoded DNS label must be 63 or fewer characters and the entire decoded DNS name is limited to 255. Only an application presenting a hostname or label consisting primarily of RFC 3454 section 3.1 "Nothing" characters to be removed would run into of this new limit. See also RFC 5894 section 6 and RFC 3491.

Update bundled libexpat to 2.5.0

The docs clearly say that PyImport_Inittab, :c:func:`PyImport_AppendInittab`, and :c:func:`PyImport_ExtendInittab` should not be used after :c:func:`Py_Initialize` has been called. We now enforce this for the two functions. Additionally, the runtime now uses an internal copy of PyImport_Inittab, to guard against modification.

Fix an issue that could potentially cause incorrect error handling for some bytecode instructions.

The compiler now removes all unused constants from code objects (except the first one, which may be a docstring).

Fix an issue that prevented :c:type:`PyThreadState` and :c:type:`PyInterpreterState` memory from being freed properly.

The 18 global C variables holding the state of the allocators have been moved to _PyRuntimeState. This is a strictly internal change with no change in behavior.

Fix failure in :keyword:`except* <except_star>` with unhashable exceptions.

Fix calculation of :data:`sys._base_executable` when inside a POSIX virtual environment using copies of the python binary when the base installation does not provide the executable name used by the venv. Calculation will fall back to alternative names ("python<MAJOR>", "python<MAJOR>.<MINOR>").

Update :mod:`faulthandler` to emit an error message with the proper unexpected signal number. Patch by Donghee Na.

Fix location of :exc:`SyntaxError` for a :keyword:`try` block with both :keyword:`except` and :keyword:`except* <except_star>`.

Merge the adaptive opcode logic into each instruction's unquickened variant, and merge the logic in EXTENDED_ARG_QUICK into :opcode:`EXTENDED_ARG`. With these changes, the quickening that happens at code object creation is now only responsible for initializing warmup counters and inserting superinstructions.

Fix the error reporting positions of specialized traceback anchors when the source line contains Unicode characters.

Improve the error suggestion for :exc:`NameError` exceptions for instances. Now if a :exc:`NameError` is raised in a method and the instance has an attribute that's exactly equal to the name in the exception, the suggestion will include self.<NAME> instead of the closest match in the method scope. Patch by Pablo Galindo

Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a :exc:`SyntaxWarning`, instead of :exc:`DeprecationWarning`. In a future Python version they will be eventually a :exc:`SyntaxError`. Patch by Victor Stinner.

A backslash-character pair that is not a valid escape sequence now generates a :exc:`SyntaxWarning`, instead of :exc:`DeprecationWarning`. For example, re.compile("\d+\.\d+") now emits a :exc:`SyntaxWarning` ("\d" is an invalid escape sequence), use raw strings for regular expression: re.compile(r"\d+\.\d+"). In a future Python version, :exc:`SyntaxError` will eventually be raised, instead of :exc:`SyntaxWarning`. Patch by Victor Stinner.

Handle StopIteration and StopAsyncIteration raised in generator or coroutines in the bytecode, rather than in wrapping C code.

Improve the :exc:`SyntaxError` error message when the user types import x from y instead of from y import x. Patch by Pablo Galindo

Fix subscription of type aliases containing bare generic types or types like :class:`~typing.TypeVar`: for example tuple[A, T][int] and tuple[TypeVar, T][int], where A is a generic type, and T is a type variable.

Lower the recursion depth for marshal on WASI to support (in-development) wasmtime 2.0.

Fix multiple crashes in debug mode when str subclasses are used instead of str itself.

Use complete source locations to simplify detection of __future__ imports which are not at the beginning of the file. Also corrects the offset in the exception raised in one case, which was off by one and impeded highlighting.

Add specialization of :opcode:`FOR_ITER` for generators. Saves multiple layers of dispatch and checking to get from the :opcode:`FOR_ITER` instruction in the caller to the :opcode:`RESUME` in the generator.

Fix source locations of :keyword:`match` sub-patterns.

Added the methods :c:func:`PyObject_Vectorcall` and :c:func:`PyObject_VectorcallMethod` to the :ref:`Limited API <stable>` along with the auxiliary macro constant :c:macro:`PY_VECTORCALL_ARGUMENTS_OFFSET`.

The availability of these functions enables more efficient PEP 590 vector calls from binary extension modules that avoid argument boxing/unboxing overheads.

Fix an issue where member descriptors (such as those for :attr:`~object.__slots__`) could behave incorrectly or crash instead of raising a :exc:`TypeError` when accessed via an instance of an invalid type.

Rather than changing :attr:`~types.CodeType.co_code`, the interpreter will now display a :exc:`RuntimeWarning` and assign :const:`None` to any fast locals that are left unbound after jumps or :keyword:`del` statements executed while tracing.

When calling into Python code from C code, through :c:func:`PyEval_EvalFrameEx` or a related C-API function, a shim frame in inserted into the call stack. This occurs in the _PyEval_EvalFrameDefault() function. The extra frame should be invisible to all Python and most C extensions, but out-of-process profilers and debuggers need to be aware of it. These shim frames can be detected by checking frame->owner == FRAME_OWNED_BY_CSTACK.

Extensions implementing their own interpreters using PEP 523 need to be aware of this shim frame and the changes to the semantics of :opcode:`RETURN_VALUE`, :opcode:`YIELD_VALUE`, and :opcode:`RETURN_GENERATOR`, which now clear the frame.

Fix detection of MAC addresses for :mod:`uuid` on certain OSs. Patch by Chaim Sanders

Quicken all code objects, and specialize adaptive bytecode instructions more aggressively.

Print exception class name instead of its string representation when raising errors from :mod:`ctypes` calls.

:exc:`ImportError` raised from failed from <module> import <name> now include suggestions for the value of <name> based on the available names in <module>. Patch by Pablo Galindo

The :opcode:`FOR_ITER` now leaves the iterator on the stack on termination of the loop. This is to assist specialization of loops for generators.

Add _pylong.py module. It includes asymptotically faster algorithms that can be used for operations on integers with many digits. It is used by longobject.c to speed up some operations.

Expose :const:`~socket.ETH_P_ALL` and some of the :ref:`ETHERTYPE_* constants <socket-ethernet-types>` in :mod:`socket`. Patch by Noam Cohen.

Allow :mod:`pdb` to locate source for frozen modules in the standard library.

Fix bug in :func:`urllib.parse.urlparse` that causes URL schemes that begin with a digit, a plus sign, or a minus sign to be parsed incorrectly.

Deprecate :class:`asyncio.AbstractChildWatcher` to be removed in Python 3.14. Patch by Kumar Aditya.

Improve performance of :func:`secrets.token_hex`.

Fixed bug where :func:`inspect.signature` reported incorrect arguments for decorated methods.

Fix SystemError in :mod:`ctypes` when exception was not set during __initsubclass__.

Remove older version of _SSLProtocolTransport.get_write_buffer_limits in :mod:`!asyncio.sslproto`

fix negative numbers failing in verify()

Fix :class:`statistics.NormalDist` pickle with 0 and 1 protocols.

enum.auto() is now correctly activated when combined with other assignment values. E.g. ONE = auto(), 'some text' will now evaluate as (1, 'some text').

Update the bundled copy of pip to version 22.3.1.

Remove the distutils package. It was deprecated in Python 3.10 by PEP 632 "Deprecate distutils module". For projects still using distutils and cannot be updated to something else, the setuptools project can be installed: it still provides distutils. Patch by Victor Stinner.

Now :mod:`_pyio` is consistent with :mod:`_io` in raising ValueError when executing methods over closed buffers.

Clean up refleak on failed module initialisation in :mod:`_zoneinfo`

Clean up refleaks on failed module initialisation in :mod:`_pickle`

Clean up refleak on failed module initialisation in :mod:`_io`.

Fix memory leak in :func:`math.dist` when both points don't have the same dimension. Patch by Kumar Aditya.

Use the frame bound builtins when offering a name suggestion in :mod:`traceback` to prevent crashing when __builtins__ is not a dict.

In :mod:`importlib._bootstrap`, enhance namespace package repr to <module 'x' (namespace) from ['path']>.

Fix _SelectorDatagramTransport to inherit from :class:`~asyncio.DatagramTransport` in :mod:`asyncio`. Patch by Kumar Aditya.

Fix argument typechecks in :func:`!_overlapped.WSAConnect` and :func:`!_overlapped.Overlapped.WSASendTo` functions.

Prevent crashing in :mod:`traceback` when retrieving the byte-offset for some source files that contain certain unicode characters.

Fix internal error in the :mod:`re` module which in very rare circumstances prevented compilation of a regular expression containing a :ref:`conditional expression <re-conditional-expression>` without the "else" branch.

Fix :meth:`asyncio.StreamWriter.drain` to call protocol.connection_lost callback only once on Windows.

Add a mutex to unittest.mock.NonCallableMock to protect concurrent access to mock attributes.

The :class:`array.array` class now supports subscripting, making it a :term:`generic type`.

Improved :class:`TypeError` message for undefined abstract methods of a :class:`abc.ABC` instance. The names of the missing methods are surrounded by single-quotes to highlight them.

Allow BUILTINS to be a valid field name for frozen dataclasses.

Make sure patch.dict() can be applied on async functions.

Remove modules :mod:`asyncore` and :mod:`asynchat`, which were deprecated by PEP 594.

Fix handling of bytes :term:`path-like objects <path-like object>` in :func:`os.ismount()`.

:mod:`ftplib`: Remove the FTP_TLS.ssl_version class attribute: use the context parameter instead. Patch by Victor Stinner

Remove the keyfile and certfile parameters from the :mod:`ftplib`, :mod:`imaplib`, :mod:`poplib` and :mod:`smtplib` modules, and the key_file, cert_file and check_hostname parameters from the :mod:`http.client` module, all deprecated since Python 3.6. Use the context parameter (ssl_context in :mod:`imaplib`) instead. Patch by Victor Stinner.

Add the :attr:`~sqlite3.Connection.autocommit` attribute to :class:`sqlite3.Connection` and the autocommit parameter to :func:`sqlite3.connect` to control PEP 249-compliant :ref:`transaction handling <sqlite3-transaction-control-autocommit>`. Patch by Erlend E. Aasland.

Fixed a race condition that could cause :func:`sysconfig.get_config_var` to incorrectly return :const:`None` in multi-threaded programs.

Fix an error when using a method of objects mocked with :func:`unittest.mock.create_autospec` after it was sealed with :func:`unittest.mock.seal` function.

:func:`shutil.copytree` now applies the ignore_dangling_symlinks argument recursively.

Add walk_up argument in :meth:`pathlib.PurePath.relative_to`.

Fix IndexError in :class:`argparse.ArgumentParser` when a store_true action is given an explicit argument.

Changes wording of docstring for :func:`pathlib.Path.iterdir`.

Update uname docs to clarify the special nature of the platform attribute and to indicate when it became late-bound.

The Python test suite now fails wit exit code 4 if no tests ran. It should help detecting typos in test names and test methods.

Fix a bug in the :mod:`typing` tests where a test relying on CPython-specific implementation details was not decorated with @cpython_only and was not skipped on other implementations.

Add tests for star-unpacking with PEP 646, and some other miscellaneous PEP 646 tests.

Added explicit coverage of Py_Initialize (and hence Py_InitializeEx) back to the embedding tests (all other embedding tests migrated to Py_InitializeFromConfig in Python 3.11)

Some C API tests were moved into the new Lib/test/test_capi/ directory.

Fix -Wimplicit-int compiler warning in :program:`configure` check for PTHREAD_SCOPE_SYSTEM.

Fix build with PYTHON_FOR_REGEN=python3.8.

Specify the full path to the source location for make docclean (needed for cross-builds).

Drop unused build dependency on readelf.

Use python3.11, if available, for regeneration and freezing.

Add new tooling, in Tools/cases_generator, to generate the interpreter switch statement from a list of opcode definitions. This only affects adding, modifying or removing instruction definitions. The instruction definitions now live in Python/bytecodes.c, in the form of a custom DSL (under development). The tooling reads this file and writes Python/generated_cases.c.h, which is then included by Python/ceval.c to provide most of the cases of the main interpreter switch.

Remove PCbuild/lib.pyproj: it's not used for anything, is only a minor convenience for Visual Studio users (who probably mostly don't even know about it), and it takes a lot of maintenance effort to keep updated.

Fix make regen-test-levenshtein for out-of-tree builds.

Don't use vendored libmpdec headers if :option:`--with-system-libmpdec` is passed to :program:`configure`. Don't use vendored libexpat headers if :option:`--with-system-expat` is passed to :program:`configure`.

Update Windows builds to zlib v1.2.13. v1.2.12 has CVE-2022-37434, but the vulnerable inflateGetHeader API is not used by Python.

Assumes that a missing DLLs directory means that standard extension modules are in the executable's directory.

Update :file:`py.exe` launcher to install 3.11 by default and 3.12 on request.

Fix the :ref:`launcher` ignoring unrecognized shebang lines instead of treating them as local paths

Update Windows installer to use SQLite 3.39.4.

Update macOS installer to SQLite 3.39.4.

The :c:macro:`Py_CLEAR`, :c:macro:`Py_SETREF` and :c:macro:`Py_XSETREF` macros now only evaluate their argument once. If the argument has side effects, these side effects are no longer duplicated. Patch by Victor Stinner.

Fix use-after-free in Py_SetPythonHome(NULL), Py_SetProgramName(NULL) and _Py_SetProgramFullPath(NULL) function calls. Issue reported by Benedikt Reinartz. Patch by Victor Stinner.

Add getbufferproc and releasebufferproc to the stable API.

Some configurable capabilities of sub-interpreters have changed. They always allow subprocesses (:mod:`subprocess`) now, whereas before subprocesses could be optionally disallowed for a sub-interpreter. Instead :func:`os.exec` can now be disallowed. Disallowing daemon threads is now supported. Disallowing all threads is still allowed, but is never done by default. Note that the optional restrictions are only available through _Py_NewInterpreterFromConfig(), which isn't a public API. They do not affect the main interpreter, nor :c:func:`Py_NewInterpreter`.

A _PyInterpreterConfig has been added and _Py_NewInterpreter() has been renamed to _Py_NewInterpreterFromConfig(). The "isolated_subinterpreters" argument is now a granular config that captures the previous behavior. Note that this is all "private" API.

Py_InitializeEx now correctly calls PyConfig_Clear after initializing the interpreter (the omission didn't cause a memory leak only because none of the dynamically allocated config fields are populated by the wrapper function)

Add :c:func:`PyFrame_GetVar` and :c:func:`PyFrame_GetVarString` functions to get a frame variable by its name. Patch by Victor Stinner.