3.11.5 Regression: Flags with custom __new__ can't be iterated

[ziima]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.11.5 (main, Aug 25 2023, 23:47:33) [GCC 12.2.0]

A clear and concise description of the bug:

A behavior of Flag changed in python 3.11.5. When Flag with custom __new__ is used in combination with functional API, it's members can't be retrieved by iteration over the class.

MWE:

from enum import IntFlag

Perm = IntFlag('Perm', {'R': 4, 'W': 2, 'X': 1})
print(tuple(Perm))


class LabeledFlag(IntFlag):
    def __new__(cls, value: int, label: str):
        obj = super().__new__(cls, value)
        obj._value_ = value
        obj.label = label
        return obj


LabeledPerm = LabeledFlag('LabeledPerm', {'R': (4, 'Read'), 'W': (2, 'Write'), 'X': (1, 'Exec')})
print(tuple(LabeledPerm))

The output in python 3.11.4:

(<Perm.R: 4>, <Perm.W: 2>, <Perm.X: 1>)
(<LabeledPerm.R: 4>, <LabeledPerm.W: 2>, <LabeledPerm.X: 1>)

The output in python 3.11.5:

(<Perm.R: 4>, <Perm.W: 2>, <Perm.X: 1>)
()

I suspect this commit to introduce the regression: 59f009e

A workaround for 3.11.5:

class LabeledFlag(IntFlag):
    def __new__(cls, value: int, label: str):
        obj = super().__new__(cls, value)
        obj._value_ = value
        obj.label = label
        return obj

    @classmethod
    def _missing_(cls, value):
        pseudo_member = super(_DnskeyFlagBase, cls)._missing_(value)
        if value in cls._value2member_map_ and pseudo_member.name is None:
            cls._value2member_map_.pop(value)
        return pseudo_member

Linked PRs

• gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ #108704
• [3.12] gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ (GH-108704) #108733
• [3.11] gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ (GH-108704) #108739

[ethanfurman]

This ended up being a combination of user error, program error, and some not-detailed-enough documentation.

An enum's custom __new__ should not use super(...).__new__(), because the parent's __new__ is only for lookups; either use the data type directly:

def __new__(cls, value, label):
    obj = int.__new__(cls, value)

or, if using an already defined enum type (i.e. LabeledFlag(IntFlag) and not LabeledFlag(int, Flag)):

def __new__(cls, value, label):
    obj = super()._new_member_(cls, value)

The code has been updated to raise a TypeError if super().__new__ is called, and a warning added to the docs.

[ethanfurman]

@ziima: the correct fix for your code:

obj = int.__new__(cls, value)

which works across all Python versions.

[ziima]

Works like a charm, thanks @ethanfurman.

I think I actually tried that, but I may had had other error in there at that moment.

[cdce8p]

• [3.12] gh-108682: [Enum] raise TypeError if super().__new__ called in custom __new__ (GH-108704) #108733

Seems to be a breaking change. Not something would have expected in an .rc2 release to be honest.
In particular raising a TypeError for enums without members worked fine before.

Haven't had the time to look at it more closely, but noticed our test suite failing once again after it had been fixed for 3.12.0rc1, especially regarding the enum changes.

• [3.12.0rc2] New enum changes zigpy/zigpy#1253

[ethanfurman]

For posterity: the issue is that zigpy has empty enums, but uses a custom _missing_ to create their members on demand. This PR broke that, but I consider that behavior to be anti-enum, and keeping it also means keeping the bug in this issue.

The solution for zigpy is to move the desired behavior into a custom metaclass.