3.6.rst

What's New In Python 3.6

Release:	|release|
Date:	|today|

This article explains the new features in Python 3.6, compared to 3.5.

For full details, see the :source:`Misc/NEWS` file.

Note

Prerelease users should be aware that this document is currently in draft form. It will be updated substantially as Python 3.6 moves towards release, so it's worth checking back even after reading earlier versions.

Summary -- Release highlights

• PEP 498: :ref:`Formatted string literals <whatsnew-fstrings>`

New Features

PEP 498: Formatted string literals

Formatted string literals are a new kind of string literal, prefixed with 'f'. They are similar to the format strings accepted by :meth:`str.format`. They contain replacement fields surrounded by curly braces. The replacement fields are expressions, which are evaluated at run time, and then formatted using the :func:`format` protocol.

>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'

See PEP 498 and the main documentation at :ref:`f-strings`.

PYTHONMALLOC environment variable

The new :envvar:`PYTHONMALLOC` environment variable allows to set the Python memory allocators and/or install debug hooks.

It is now possible to install debug hooks on Python memory allocators on Python compiled in release mode using PYTHONMALLOC=debug. Effects of debug hooks:

• Newly allocated memory is filled with the byte 0xCB
• Freed memory is filled with the byte 0xDB
• Detect violations of Python memory allocator API. For example, :c:func:`PyObject_Free` called on a memory block allocated by :c:func:`PyMem_Malloc`.
• Detect write before the start of the buffer (buffer underflow)
• Detect write after the end of the buffer (buffer overflow)
• Check that the :term:`GIL <global interpreter lock>` is held when allocator functions of :c:data:`PYMEM_DOMAIN_OBJ` (ex: :c:func:`PyObject_Malloc`) and :c:data:`PYMEM_DOMAIN_MEM` (ex: :c:func:`PyMem_Malloc`) domains are called.

Checking if the GIL is held is also a new feature of Python 3.6.

See the :c:func:`PyMem_SetupDebugHooks` function for debug hooks on Python memory allocators.

It is now also possible to force the usage of the :c:func:`malloc` allocator of the C library for all Python memory allocations using PYTHONMALLOC=malloc. It helps to use external memory debuggers like Valgrind on a Python compiled in release mode.

On error, the debug hooks on Python memory allocators now use the :mod:`tracemalloc` module to get the traceback where a memory block was allocated.

Example of fatal error on buffer overflow using python3.6 -X tracemalloc=5 (store 5 frames in traces):

Debug memory block at address p=0x7fbcd41666f8: API 'o'
    4 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x02 *** OUCH
        at tail+1: 0xfb
        at tail+2: 0xfb
        at tail+3: 0xfb
        at tail+4: 0xfb
        at tail+5: 0xfb
        at tail+6: 0xfb
        at tail+7: 0xfb
    The block was made by call #1233329 to debug malloc/realloc.
    Data at p: 1a 2b 30 00

Memory block allocated at (most recent call first):
  File "test/test_bytes.py", line 323
  File "unittest/case.py", line 600
  File "unittest/case.py", line 648
  File "unittest/suite.py", line 122
  File "unittest/suite.py", line 84

Fatal Python error: bad trailing pad byte

Current thread 0x00007fbcdbd32700 (most recent call first):
  File "test/test_bytes.py", line 323 in test_hex
  File "unittest/case.py", line 600 in run
  File "unittest/case.py", line 648 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  ...

(Contributed by Victor Stinner in :issue:`26516` and :issue:`26564`.)

Other Language Changes

• None yet.

New Modules

• None yet.

Improved Modules

contextlib

The :class:`contextlib.AbstractContextManager` class has been added to provide an abstract base class for context managers. It provides a sensible default implementation for __enter__() which returns self and leaves __exit__() an abstract method. A matching class has been added to the :mod:`typing` module as :class:`typing.ContextManager`. (Contributed by Brett Cannon in :issue:`25609`.)

datetime

The :meth:`datetime.strftime() <datetime.datetime.strftime>` and :meth:`date.strftime() <datetime.date.strftime>` methods now support ISO 8601 date directives %G, %u and %V. (Contributed by Ashley Anderson in :issue:`12006`.)

faulthandler

On Windows, the :mod:`faulthandler` module now installs an handler for Windows exceptions: see :func:`faulthandler.enable`. (Contributed by Victor Stinner in :issue:`23848`.)

os

A new :meth:`~os.scandir.close` method allows explicitly closing a :func:`~os.scandir` iterator. The :func:`~os.scandir` iterator now supports the :term:`context manager` protocol. If a :func:`scandir` iterator is neither exhausted nor explicitly closed a :exc:`ResourceWarning` will be emitted in its destructor. (Contributed by Serhiy Storchaka in :issue:`25994`.)

pickle

Objects that need calling __new__ with keyword arguments can now be pickled using :ref:`pickle protocols <pickle-protocols>` older than protocol version 4. Protocol version 4 already supports this case. (Contributed by Serhiy Storchaka in :issue:`24164`.)

rlcompleter

Private and special attribute names now are omitted unless the prefix starts with underscores. A space or a colon is added after some completed keywords. (Contributed by Serhiy Storchaka in :issue:`25011` and :issue:`25209`.)

Names of most attributes listed by :func:`dir` are now completed. Previously, names of properties and slots which were not yet created on an instance were excluded. (Contributed by Martin Panter in :issue:`25590`.)

site

When specifying paths to add to :attr:`sys.path` in a .pth file, you may now specify file paths on top of directories (e.g. zip files). (Contributed by Wolfgang Langner in :issue:`26587`).

telnetlib

:class:`~telnetlib.Telnet` is now a context manager (contributed by Stéphane Wirtel in :issue:`25485`).

typing

The :class:`typing.ContextManager` class has been added for representing :class:`contextlib.AbstractContextManager`. (Contributed by Brett Cannon in :issue:`25609`.)

unittest.mock

The :class:`~unittest.mock.Mock` class has the following improvements:

• Two new methods, :meth:`Mock.assert_called() <unittest.mock.Mock.assert_called>` and :meth:`Mock.assert_called_once() <unittest.mock.Mock.assert_called_once>` to check if the mock object was called. (Contributed by Amit Saha in :issue:`26323`.)

urllib.robotparser

:class:`~urllib.robotparser.RobotFileParser` now supports the Crawl-delay and Request-rate extensions. (Contributed by Nikolay Bogoychev in :issue:`16099`.)

warnings

A new optional source parameter has been added to the :func:`warnings.warn_explicit` function: the destroyed object which emitted a :exc:`ResourceWarning`. A source attribute has also been added to :class:`warnings.WarningMessage` (contributed by Victor Stinner in :issue:`26568` and :issue:`26567`).

When a :exc:`ResourceWarning` warning is logged, the :mod:`tracemalloc` is now used to try to retrieve the traceback where the detroyed object was allocated.

Example with the script example.py:

import warnings

def func():
    return open(__file__)

f = func()
f = None

Output of the command python3.6 -Wd -X tracemalloc=5 example.py:

example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'>
  f = None
Object allocated at (most recent call first):
  File "example.py", lineno 4
    return open(__file__)
  File "example.py", lineno 6
    f = func()

The "Object allocated at" traceback is new and only displayed if :mod:`tracemalloc` is tracing Python memory allocations and if the :mod:`warnings` was already imported.

zipfile

A new :meth:`ZipInfo.from_file() <zipfile.ZipInfo.from_file>` class method allows making a :class:`~zipfile.ZipInfo` instance from a filesystem file. A new :meth:`ZipInfo.is_dir() <zipfile.ZipInfo.is_dir>` method can be used to check if the :class:`~zipfile.ZipInfo` instance represents a directory. (Contributed by Thomas Kluyver in :issue:`26039`.)

zlib

The :func:`~zlib.compress` function now accepts keyword arguments. (Contributed by Aviv Palivoda in :issue:`26243`.)

Optimizations

• The ASCII decoder is now up to 60 times as fast for error handlers surrogateescape, ignore and replace (Contributed by Victor Stinner in :issue:`24870`).
• The ASCII and the Latin1 encoders are now up to 3 times as fast for the error handler surrogateescape (Contributed by Victor Stinner in :issue:`25227`).
• The UTF-8 encoder is now up to 75 times as fast for error handlers ignore, replace, surrogateescape, surrogatepass (Contributed by Victor Stinner in :issue:`25267`).
• The UTF-8 decoder is now up to 15 times as fast for error handlers ignore, replace and surrogateescape (Contributed by Victor Stinner in :issue:`25301`).
• bytes % args is now up to 2 times faster. (Contributed by Victor Stinner in :issue:`25349`).
• bytearray % args is now between 2.5 and 5 times faster. (Contributed by Victor Stinner in :issue:`25399`).
• Optimize :meth:`bytes.fromhex` and :meth:`bytearray.fromhex`: they are now between 2x and 3.5x faster. (Contributed by Victor Stinner in :issue:`25401`).
• Optimize bytes.replace(b'', b'.') and bytearray.replace(b'', b'.'): up to 80% faster. (Contributed by Josh Snider in :issue:`26574`).

Build and C API Changes

• New :c:func:`Py_FinalizeEx` API which indicates if flushing buffered data failed (:issue:`5319`).

Deprecated

New Keywords

async and await are not recommended to be used as variable, class, function or module names. Introduced by PEP 492 in Python 3.5, they will become proper keywords in Python 3.7.

Deprecated Python modules, functions and methods

• :meth:`importlib.machinery.SourceFileLoader.load_module` and :meth:`importlib.machinery.SourcelessFileLoader.load_module` are now deprecated. They were the only remaining implementations of :meth:`importlib.abc.Loader.load_module` in :mod:`importlib` that had not been deprecated in previous versions of Python in favour of :meth:`importlib.abc.Loader.exec_module`.

Deprecated functions and types of the C API

• None yet.

Deprecated features

• The pyvenv script has been deprecated in favour of python3 -m venv. This prevents confusion as to what Python interpreter pyvenv is connected to and thus what Python interpreter will be used by the virtual environment. (Contributed by Brett Cannon in :issue:`25154`.)
• When performing a relative import, falling back on __name__ and __path__ from the calling module when __spec__ or __package__ are not defined now raises an :exc:`ImportWarning`. (Contributed by Rose Ames in :issue:`25791`.)

Deprecated Python behavior

• Raising the :exc:`StopIteration` exception inside a generator will now generate a :exc:`DeprecationWarning`, and will trigger a :exc:`RuntimeError` in Python 3.7. See :ref:`whatsnew-pep-479` for details.

Removed

API and Feature Removals

• inspect.getmoduleinfo() was removed (was deprecated since CPython 3.3). :func:`inspect.getmodulename` should be used for obtaining the module name for a given path.
• traceback.Ignore class and traceback.usage, traceback.modname, traceback.fullmodname, traceback.find_lines_from_code, traceback.find_lines, traceback.find_strings, traceback.find_executable_lines methods were removed from the :mod:`traceback` module. They were undocumented methods deprecated since Python 3.2 and equivalent functionality is available from private methods.

Porting to Python 3.6

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in the Python API

• The format of the co_lnotab attribute of code objects changed to support negative line number delta. By default, Python does not emit bytecode with negative line number delta. Functions using frame.f_lineno, PyFrame_GetLineNumber() or PyCode_Addr2Line() are not affected. Functions decoding directly co_lnotab should be updated to use a signed 8-bit integer type for the line number delta, but it's only required to support applications using negative line number delta. See Objects/lnotab_notes.txt for the co_lnotab format and how to decode it, and see the PEP 511 for the rationale.
• The functions in the :mod:`compileall` module now return booleans instead of 1 or 0 to represent success or failure, respectively. Thanks to booleans being a subclass of integers, this should only be an issue if you were doing identity checks for 1 or 0. See :issue:`25768`.
• Reading the :attr:`~urllib.parse.SplitResult.port` attribute of :func:`urllib.parse.urlsplit` and :func:`~urllib.parse.urlparse` results now raises :exc:`ValueError` for out-of-range values, rather than returning :const:`None`. See :issue:`20059`.
• The :mod:`imp` module now raises a :exc:`DeprecationWarning` instead of :exc:`PendingDeprecationWarning`.
• The following modules have had missing APIs added to their :attr:`__all__` attributes to match the documented APIs: :mod:`calendar`, :mod:`csv`, :mod:`~xml.etree.ElementTree`, :mod:`enum`, :mod:`fileinput`, :mod:`ftplib`, :mod:`logging`, :mod:`optparse`, :mod:`tarfile`, :mod:`threading` and :mod:`wave`. This means they will export new symbols when import * is used. See :issue:`23883`.
• When performing a relative import, if __package__ does not compare equal to __spec__.parent then :exc:`ImportWarning` is raised. (Contributed by Brett Cannon in :issue:`25791`.)
• When a relative import is performed and no parent package is known, then :exc:`ImportError` will be raised. Previously, :exc:`SystemError` could be raised. (Contributed by Brett Cannon in :issue:`18018`.)
• Servers based on the :mod:`socketserver` module, including those defined in :mod:`http.server`, :mod:`xmlrpc.server` and :mod:`wsgiref.simple_server`, now only catch exceptions derived from :exc:`Exception`. Therefore if a request handler raises an exception like :exc:`SystemExit` or :exc:`KeyboardInterrupt`, :meth:`~socketserver.BaseServer.handle_error` is no longer called, and the exception will stop a single-threaded server. (Contributed by Martin Panter in :issue:`23430`.)
• :func:`spwd.getspnam` now raises a :exc:`PermissionError` instead of :exc:`KeyError` if the user doesn't have privileges.
• The :meth:`socket.socket.close` method now raises an exception if an error (e.g. EBADF) was reported by the underlying system call. See :issue:`26685`.

Changes in the C API

• :c:func:`Py_Exit` (and the main interpreter) now override the exit status with 120 if flushing buffered data failed. See :issue:`5319`.