bpo-44466: Faulthandler now detects the GC

[vstinner]

The faulthandler module now detects if a fatal error occurs during a
garbage collector collection (only if all_threads is true).

https://bugs.python.org/issue44466

[vstinner]

@pablogsal: We are past the Python 3.10 feature freeze. Since faulthandler is a debug module and I consider that the traceback.c change is trivial, can I please get an exception to get this feature in Python 3.10?

[pablogsal]

Please, are two more tests:

• gc running on a different thread
• gc not running

[vstinner]

The "gc not running" case is covered by all other tests in test_faulthandler. The file contains 45 tests.

[vstinner]

The "gc not running" case is covered by all other tests in test_faulthandler.

The check_error() helper function in tests uses a strict regular expression matching the exact output. It fails if Garbage-collecting is found of the output.

[vstinner]

@pablogsal:

Please, are two more tests: (...) gc running on a different thread

Such test would require to ensure that thread A is running the GC and thread B triggers faulthandler somehow. The problem is that the GIL prevents to run Python code in thread B (since thread A holds the GIL to run the GC). Thread B could use C code and a threading event to wait until thread A is running the GC (and then thread A must wait until the fatal error occurs). But I would prefer to not have to write C code to wait for an event set in Python: threading.Event.wait() is implemented in pure Python.

Moreover, faulthandler and test_faulthandler are already very fragile and painful to maintain. I would prefer to avoid adding a new functional test which requires to synchronize two threads. In my experience, faulthandler is already failing for more stupid reasons, like a test failing because the child process output is empty and I have no way to debug such issue. It looks random and I don't have access to the buildbot worker where it happens.

[vstinner]

Since this change is done in the _Py_DumpTracebackThreads() function and not directly in the faulthandler module, the nice side effect is that the Py_FatalError() function automatically inherits this new feature!

Example:

Fatal Python error: test_fatal_error: bug
Python runtime state: initialized

Current thread 0x00007f80c8f73740 (most recent call first):
  Garbage-collecting
  File "/home/vstinner/python/main/x.py", line 11 in __del__
  File "/home/vstinner/python/main/x.py", line 25 in <module>

Extension modules: _testcapi (total: 1)
Abandon (core dumped)

With code:

import _testcapi
import faulthandler
import gc
import sys

faulthandler.enable()

class RefCycle:
    def __del__(self):
        _testcapi.fatal_error(b'bug')

# create a reference cycle which triggers a fatal
# error in a destructor
a = RefCycle()
b = RefCycle()
a.b = b
b.a = a

# Delete the objects, not the cycle
a = None
b = None

# Break the reference cycle: call __del__()
gc.collect()

# Should not reach this line
print("exit", file=sys.stderr)

[miss-islington]

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

[miss-islington]

Sorry @vstinner, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker d19163912bfc790283724f05328bd31e4e65003d 3.10

[miss-islington]

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

[miss-islington]

Sorry @vstinner, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker d19163912bfc790283724f05328bd31e4e65003d 3.10

[bedevere-bot]

GH-26826 is a backport of this pull request to the 3.10 branch.

[vstinner]

I don't understand why @miss-islington failed to create the backport PR. Locally, git cherry-pick -x worked without any conflict. I created the backport PR manually.