Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_syslog: test_syslog_threaded() crashs randomly on ARM64 macOS 3.x #98178

Closed
vstinner opened this issue Oct 11, 2022 · 31 comments
Closed

test_syslog: test_syslog_threaded() crashs randomly on ARM64 macOS 3.x #98178

vstinner opened this issue Oct 11, 2022 · 31 comments
Labels
type-bug An unexpected behavior, bug, or error

Comments

@vstinner
Copy link
Member

test_syslog crashed 4 times in the last 2 days, each time in the test_syslog_threaded() function, on the "ARM64 macOS 3.x" buildbot:

Example of crash:

Fatal Python error: Segmentation fault

Thread 0x000000017bbcf000 (most recent call first):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_syslog.py", line 67 in logger
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 986 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 1049 in _bootstrap_inner
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.pymake: *** [buildbottest] Segmentation fault: 11

This crash is on the syslog.syslog() call in a thread. Code:

    @threading_helper.requires_working_threading()
    def test_syslog_threaded(self):
        start = threading.Event()
        stop = False
        def opener():
            start.wait(10)
            i = 1
            while not stop:
                syslog.openlog(f'python-test-{i}')  # new string object
                i += 1
        def logger():
            start.wait(10)
            while not stop:
                syslog.syslog('test message from python test_syslog') # <=== HERE: line 67

        orig_si = sys.getswitchinterval()
        support.setswitchinterval(1e-9)
        try:
            threads = [threading.Thread(target=opener)]
            threads += [threading.Thread(target=logger) for k in range(10)]
            with threading_helper.start_threads(threads):
                start.set()
                time.sleep(0.1)
                stop = True
        finally:
            sys.setswitchinterval(orig_si)
@vstinner vstinner added the type-bug An unexpected behavior, bug, or error label Oct 11, 2022
@vstinner
Copy link
Member Author

on the "ARM64 macOS 3.x" buildbot

It's running macOS 11.6 / ARM64 (Aarch64): Apple M1 CPU.

test.pythoninfo:

build.NDEBUG: build assertions (macro not defined)
build.Py_DEBUG: Yes (sys.gettotalrefcount() present)

datetime.datetime.now: 2022-10-11 04:29:08.870490

os.environ[MACOSX_DEPLOYMENT_TARGET]: 11.6
os.environ[TMPDIR]: /var/folders/39/qz2m3x352hd7djhs80p6vf7c0000gp/T/
os.getcwd: /Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build

platform.architecture: 64bit
platform.platform: macOS-11.6-arm64-arm-64bit
platform.python_implementation: CPython

sys.version: 3.12.0a0 (heads/main:b399115ef1, Oct 11 2022, 04:28:51) [Clang 13.0.0 (clang-1300.0.29.3)]

sysconfig[HOST_GNU_TYPE]: aarch64-apple-darwin20.6.0

@vstinner
Copy link
Member Author

@vstinner
Copy link
Member Author

The crash started to occur at build 2780 which was triggered by b399115 : no idea if it's related to this change.

@vstinner
Copy link
Member Author

It's unclear to me if the 3.11 branch is affected on this buildbot worker (ARM64 macOS). There was a single build on the 3.11 branch in the last 2 days (since the crash started to occur on the main branch): https://buildbot.python.org/all/#/builders/1030 I don't see any crash on the 3.11 branch for one month.

@davfsa
Copy link

davfsa commented Oct 11, 2022

Something else that might be useful to note is that both crashes you linked to were running test_multiprocessing_spawn at the same time as the crash occurred.

https://buildbot.python.org/all/#/builders/725/builds/2780/steps/5/logs/stdio (line 297)

0:01:56 load avg: 3.14 [106/434/1] test_syslog crashed (Exit code -11) -- running: test_multiprocessing_spawn (1 min 37 sec), test_asyncio (33.1 sec), test_concurrent_futures (1 min 28 sec)

https://buildbot.python.org/all/#/builders/725/builds/2786/steps/5/logs/stdio (line 627)

0:05:50 load avg: 2.50 [389/434/1] test_syslog crashed (Exit code -11) -- running: test_multiprocessing_spawn (1 min 14 sec)

Which doesnt seem to be the case for other non-crashing builds (ie https://buildbot.python.org/all/#/builders/725/builds/2785/steps/5/logs/stdio)

Could these be related?

@Fidget-Spinner
Copy link
Member

Fidget-Spinner commented Oct 11, 2022

b399115 is not the first crash. See https://buildbot.python.org/all/#/builders/725.

First crash in recent times was 2 days ago https://buildbot.python.org/all/#/builders/725/builds/2780 during testing of commit d876528.

@ambv
Copy link
Contributor

ambv commented Oct 11, 2022

I'm unable to reproduce neither with pydebug nor with an opt build, neither with -F nor with re-starting -m test thousands of times. Now I'm thinking this has to be an interdependence between different tests. The obvious candidates are test_thread* but I'm somehow thinking this will be test_logging, test_asyncio, or test_concurrent_futures. Trying with all those to find the culprit right now... no dice yet.

@ambv
Copy link
Contributor

ambv commented Oct 11, 2022

Doesn't seem to be multiprocessing (although one of those runs is deadlocked):

1:00:09 load avg: 26.97 [358] test_syslog passed -- running: test_multiprocessing_spawn (32.1 sec), test_multiprocessing_spawn (51 min 18 sec), test_multiprocessing_spawn (1 min 4 sec), test_multiprocessing_spawn (1 min 11 sec), test_multiprocessing_spawn (30.3 sec)

@ronaldoussoren
Copy link
Contributor

A crash log from the buildbot would be nice, that should contain a stack trace of the crash.

BTW. cae7d1d recently added more syslog tests. That commit also has some refcounting fixes in _testcapimodule.c, which look sane to me.

@ambv
Copy link
Contributor

ambv commented Oct 11, 2022

Also doesn't look like it's concurrent.futures related...

1:19:15 load avg: 10.72 [378] test_syslog passed -- running: test_concurrent_futures (1 min 5 sec), test_concurrent_futures (41.6 sec)

I'm in the process of getting the crash log from the box for you, Ronald.

@ned-deily
Copy link
Member

Do we know for sure that the libc syslog on macOS is thread-safe? There seems to be some discussion in the past about this on the webs.

@ronaldoussoren
Copy link
Contributor

Do we know for sure that the libc syslog on macOS is thread-safe? There seems to be some discussion in the past about this on the webs.

Indeed. This issue mentions ASL handles cannot be used concurrently on two threads and the syslog implementation uses a single global ASL handle. The most recent code dump from Apple uses a lock to create the ASL handle, but doesn't guard usage of said handle.

This may well be an OS bug.

Which begs the question: Is this something we want to work around by introducing a lock on our end, even if that isn't a 100% fix because this wouldn't guard syslog() calls from C code. If not, disabling the multi-threading test in test_syslog would likely avoid the crash by not calling syslog concurrently on our end.

Sigh...

@ambv
Copy link
Contributor

ambv commented Oct 11, 2022

@ronaldoussoren unfortunately it seems that buildbot python.exe crashes don't produce any output in our buildbot's ~/Library/Logs/DiagnosticReports. @pablogsal checked that manually producing a C binary with a bus error and running it does put a file in this folder. Somehow not when we're running automated builds. Since he's out on vacation, we'll continue the hunt after he's back. I was unable to reproduce today on macOS 12.6. The buildbot runs 11.6, maybe that's the differentiating factor.

@ronaldoussoren
Copy link
Contributor

BTW. I haven't been able to reproduce the issue on my end, but haven't tried tested on macOS 11 yet (the version used on the buildbot).

@ned-deily
Copy link
Member

unfortunately it seems that buildbot python.exe crashes don't produce any output in our buildbot's ~/Library/Logs/DiagnosticReports

I believe it is the case that those crash reports aren't produced by macOS for non-gui programs, i.e if Python is not running under an app bundle. A Python "framework build" fixes that.

@ned-deily
Copy link
Member

er, if so, that would need to be running from an installed framework build, so something like ./configure --enable-framework=$HOME/Library/Frameworks --with-pydebug ... && make -j4 && make install

@vstinner
Copy link
Member Author

vstinner commented Oct 12, 2022

We already have such kind of lock: Modules/socketmodule.c uses:

/* Lock to allow python interpreter to continue, but only allow one
   thread to be in gethostbyname or getaddrinfo */
#if defined(USE_GETHOSTBYNAME_LOCK) || defined(USE_GETADDRINFO_LOCK)
static PyThread_type_lock netdb_lock;
#endif

gethostbyname() and getaddrinfo() are no thread-safe on some platforms.

@vstinner
Copy link
Member Author

Another simple option is to not release the GIL on macOS: I created PR #98213 for that.

@corona10
Copy link
Member

Another simple option is to not release the GIL on macOS: I created PR #98213 for that.

I prefer this option.

@vstinner
Copy link
Member Author

@pablogsal
Copy link
Member

I can reproduce this easily on the buidbot with ./python.exe -m test test_syslog -F. Unfortunately i cannot attach lldb because reasons;

pablogsal@moonstone cpython2 % lldb -- ./python.exe -m test test_dbm -m test_anydbm_creation
(lldb) target create "./python.exe"
Current executable set to '/Users/pablogsal/cpython2/python.exe' (arm64).
(lldb) settings set -- target.run-args  "-m" "test" "test_dbm" "-m" "test_anydbm_creation"
(lldb) r
error: process exited with status -1 (developer mode is not enabled on this machine and this is a non-interactive debug session.)

🤷

But OTOH i managed to get a crash file by crashing it by hand:


Process:               python.exe [35552]
Path:                  /Users/USER/*/python.exe
Identifier:            python.exe
Version:               0
Code Type:             ARM-64 (Native)
Parent Process:        ??? [933]
User ID:               502

Date/Time:             2022-10-12 22:50:47.829 +0100
OS Version:            macOS 11.6 (20G165)
Report Version:        12
Anonymous UUID:        A90317BC-0225-F12B-18B3-8EDC1B000CCE


Time Awake Since Boot: 130000 seconds

System Integrity Protection: enabled

Crashed Thread:        9

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0xffffffffffffffff
Exception Note:        EXC_CORPSE_NOTIFY

VM Regions Near 0xffffffffffffffff:
--> commpage (reserved)        1000000000-7000000000   [384.0G] ---/--- SM=NUL  reserved VM address space (unallocated)

Thread 0:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib              0x000000019cbd3a48 __psynch_mutexwait + 8
1   libsystem_pthread.dylib             0x000000019cc08918 _pthread_mutex_firstfit_lock_wait + 88
2   libsystem_pthread.dylib             0x000000019cc06244 _pthread_mutex_firstfit_lock_slow + 232
3   libsystem_pthread.dylib             0x000000019cc0be3c _pthread_cond_wait + 1392
4   python.exe                          0x0000000100388394 PyCOND_TIMEDWAIT + 52 (condvar.h:73)
5   python.exe                          0x0000000100387584 take_gil + 180 (ceval_gil.c:380)
6   python.exe                          0x0000000100387ca0 PyEval_RestoreThread + 40 (ceval_gil.c:661)
7   python.exe                          0x000000010041eae0 pysleep + 132 (timemodule.c:2166)
8   python.exe                          0x000000010041e088 time_sleep + 56 (timemodule.c:383)
9   python.exe                          0x00000001002d1b18 cfunction_vectorcall_O + 148 (methodobject.c:514)
10  python.exe                          0x000000010028c840 _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
11  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
12  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
13  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
14  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
15  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
16  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
17  python.exe                          0x000000010028f3f8 method_vectorcall + 308 (classobject.c:89)
18  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
19  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
20  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
21  python.exe                          0x0000000100363d60 do_call_core + 532
22  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
23  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
24  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
25  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
26  python.exe                          0x000000010028c990 _PyObject_FastCallDictTstate + 240 (call.c:142)
27  python.exe                          0x000000010028d7ac _PyObject_Call_Prepend + 196 (call.c:485)
28  python.exe                          0x00000001002eca70 slot_tp_call + 112 (typeobject.c:8089)
29  python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
30  python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
31  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
32  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
33  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
34  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
35  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
36  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
37  python.exe                          0x000000010028f3f8 method_vectorcall + 308 (classobject.c:89)
38  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
39  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
40  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
41  python.exe                          0x0000000100363d60 do_call_core + 532
42  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
43  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
44  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
45  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
46  python.exe                          0x000000010028c990 _PyObject_FastCallDictTstate + 240 (call.c:142)
47  python.exe                          0x000000010028d7ac _PyObject_Call_Prepend + 196 (call.c:485)
48  python.exe                          0x00000001002eca70 slot_tp_call + 112 (typeobject.c:8089)                                                                                                                                                                                                                                                                                                                                      49  python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
50  python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
51  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
52  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
53  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
54  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
55  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
56  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
57  python.exe                          0x000000010028f3f8 method_vectorcall + 308 (classobject.c:89)
58  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
59  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
60  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
61  python.exe                          0x0000000100363d60 do_call_core + 532
62  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
63  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
64  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
65  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
66  python.exe                          0x000000010028c990 _PyObject_FastCallDictTstate + 240 (call.c:142)
67  python.exe                          0x000000010028d7ac _PyObject_Call_Prepend + 196 (call.c:485)
68  python.exe                          0x00000001002eca70 slot_tp_call + 112 (typeobject.c:8089)
69  python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
70  python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
71  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
72  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
73  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
74  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
75  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
76  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
77  python.exe                          0x000000010028f3f8 method_vectorcall + 308 (classobject.c:89)
78  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
79  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
80  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
81  python.exe                          0x0000000100363d60 do_call_core + 532
82  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
83  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
84  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
85  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
86  python.exe                          0x000000010028c990 _PyObject_FastCallDictTstate + 240 (call.c:142)
87  python.exe                          0x000000010028d7ac _PyObject_Call_Prepend + 196 (call.c:485)
88  python.exe                          0x00000001002eca70 slot_tp_call + 112 (typeobject.c:8089)
89  python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
90  python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
91  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
92  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
93  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
94  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
95  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
96  python.exe                          0x00000001004225f4 _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
97  python.exe                          0x0000000100423288 partial_vectorcall + 336 (_functoolsmodule.c:231)
98  python.exe                          0x000000010028c840 _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
99  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
100 python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
101 python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
102 python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
103 python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
104 python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
105 python.exe                          0x000000010028f378 method_vectorcall + 180 (classobject.c:59)
106 python.exe                          0x000000010028d158 _PyVectorcall_Call + 152 (call.c:258)
107 python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
108 python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
109 python.exe                          0x0000000100363d60 do_call_core + 532
110 python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
111 python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
112 python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
113 python.exe                          0x00000001003545a0 PyEval_EvalCode + 124 (ceval.c:583)
114 python.exe                          0x0000000100352ad8 builtin_exec_impl + 832 (bltinmodule.c:1079)
115 python.exe                          0x00000001003511a0 builtin_exec + 260 (bltinmodule.c.h:543)
116 python.exe                          0x00000001002d1978 cfunction_vectorcall_FASTCALL_KEYWORDS + 96 (methodobject.c:443)
117 python.exe                          0x000000010028c840 _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
118 python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
119 python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
120 python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
121 python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
122 python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
123 python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
124 python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
125 python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
126 python.exe                          0x00000001003cef88 pymain_run_module + 200 (main.c:300)
127 python.exe                          0x00000001003ce920 pymain_run_python + 204 (main.c:604)
128 python.exe                          0x00000001003ce814 Py_RunMain + 24 (main.c:689)
129 python.exe                          0x00000001003cea8c pymain_main + 52 (main.c:719)
130 python.exe                          0x00000001003cead0 Py_BytesMain + 40 (main.c:743)
131 python.exe                          0x000000010020f50c main + 12 (python.c:15)
132 libdyld.dylib                       0x000000019cc29430 start + 4

Thread 1:
0   python.exe                          0x00000001002d913c _PyMem_DebugFree + 4 (obmalloc.c:2707)
1   python.exe                          0x00000001002d9738 PyObject_Free + 32 (obmalloc.c:742)
2   python.exe                          0x000000010030d0c0 unicode_dealloc + 216 (unicodeobject.c:1557)
3   python.exe                          0x00000001002d8220 _Py_Dealloc + 76 (object.c:2383)
4   python.exe                          0x000000010035474c Py_DECREF + 68 (object.h:549)
5   python.exe                          0x00000001003597ec _PyEval_EvalFrameDefault + 20352 (ceval.c:2676)
6   python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
7   python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
8   python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
9   python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
10  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
11  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
12  python.exe                          0x0000000100363d60 do_call_core + 532
13  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
14  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
15  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
16  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
17  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
18  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
19  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
20  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
21  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
22  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
23  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
24  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
25  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 2:
0   libsystem_kernel.dylib              0x000000019cbd4548 __psynch_cvwait + 8
1   libsystem_pthread.dylib             0x000000019cc0bdac _pthread_cond_wait + 1248
2   python.exe                          0x0000000100388394 PyCOND_TIMEDWAIT + 52 (condvar.h:73)
3   python.exe                          0x0000000100387584 take_gil + 180 (ceval_gil.c:380)
4   python.exe                          0x0000000100387ca0 PyEval_RestoreThread + 40 (ceval_gil.c:661)
5   syslog.cpython-312d-darwin.so       0x0000000101483334 syslog_syslog_impl + 160 (syslogmodule.c:212)
6   syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
7   python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
8   python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
9   python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
10  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
11  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
12  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
13  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
14  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
15  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
16  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
17  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
18  python.exe                          0x0000000100363d60 do_call_core + 532
19  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
20  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
21  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
22  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
23  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
24  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
25  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
26  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
27  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
28  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
29  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
30  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
31  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 3:
0   libsystem_kernel.dylib              0x000000019cbd4548 __psynch_cvwait + 8
1   libsystem_pthread.dylib             0x000000019cc0bdac _pthread_cond_wait + 1248
2   python.exe                          0x0000000100388394 PyCOND_TIMEDWAIT + 52 (condvar.h:73)
3   python.exe                          0x0000000100387584 take_gil + 180 (ceval_gil.c:380)
4   python.exe                          0x0000000100387ca0 PyEval_RestoreThread + 40 (ceval_gil.c:661)
5   syslog.cpython-312d-darwin.so       0x0000000101483334 syslog_syslog_impl + 160 (syslogmodule.c:212)
6   syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
7   python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
8   python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
9   python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
10  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
11  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
12  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
13  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
14  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
15  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
16  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
17  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
18  python.exe                          0x0000000100363d60 do_call_core + 532
19  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
20  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
21  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
22  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
23  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
24  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
25  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
26  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
27  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
28  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
29  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
30  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
31  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 4:
0   libsystem_kernel.dylib              0x000000019cbd4548 __psynch_cvwait + 8
1   libsystem_pthread.dylib             0x000000019cc0bdac _pthread_cond_wait + 1248
2   python.exe                          0x0000000100388394 PyCOND_TIMEDWAIT + 52 (condvar.h:73)
3   python.exe                          0x0000000100387584 take_gil + 180 (ceval_gil.c:380)
4   python.exe                          0x0000000100387ca0 PyEval_RestoreThread + 40 (ceval_gil.c:661)
5   syslog.cpython-312d-darwin.so       0x0000000101483334 syslog_syslog_impl + 160 (syslogmodule.c:212)
6   syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
7   python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
8   python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
9   python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
10  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
11  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
12  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
13  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
14  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
15  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
16  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
17  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
18  python.exe                          0x0000000100363d60 do_call_core + 532
19  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
20  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
21  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
22  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
23  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
24  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
25  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
26  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
27  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
28  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
29  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
30  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
31  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 5:
0   libsystem_kernel.dylib              0x000000019cbd4548 __psynch_cvwait + 8
1   libsystem_pthread.dylib             0x000000019cc0bdac _pthread_cond_wait + 1248
2   python.exe                          0x0000000100388394 PyCOND_TIMEDWAIT + 52 (condvar.h:73)
3   python.exe                          0x0000000100387584 take_gil + 180 (ceval_gil.c:380)
4   python.exe                          0x0000000100387ca0 PyEval_RestoreThread + 40 (ceval_gil.c:661)
5   syslog.cpython-312d-darwin.so       0x0000000101483334 syslog_syslog_impl + 160 (syslogmodule.c:212)
6   syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
7   python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
8   python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
9   python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
10  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
11  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
12  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
13  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
14  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
15  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
16  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
17  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
18  python.exe                          0x0000000100363d60 do_call_core + 532
19  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
20  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
21  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
22  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
23  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
24  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
25  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
26  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
27  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
28  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
29  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
30  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
31  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 6:
0   libsystem_kernel.dylib              0x000000019cbd4548 __psynch_cvwait + 8
1   libsystem_pthread.dylib             0x000000019cc0bdac _pthread_cond_wait + 1248
2   python.exe                          0x0000000100388394 PyCOND_TIMEDWAIT + 52 (condvar.h:73)
3   python.exe                          0x0000000100387584 take_gil + 180 (ceval_gil.c:380)
4   python.exe                          0x0000000100387ca0 PyEval_RestoreThread + 40 (ceval_gil.c:661)
5   syslog.cpython-312d-darwin.so       0x0000000101483334 syslog_syslog_impl + 160 (syslogmodule.c:212)
6   syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
7   python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
8   python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
9   python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
10  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
11  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
12  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
13  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
14  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
15  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
16  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
17  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
18  python.exe                          0x0000000100363d60 do_call_core + 532
19  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
20  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
21  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
22  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
23  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
24  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
25  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
26  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
27  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
28  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
29  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
30  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
31  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 7:                                                                                                                                                                                                                                                                                                                                                                                                                    [931/5095]0   libsystem_kernel.dylib              0x000000019cbd4548 __psynch_cvwait + 8
1   libsystem_pthread.dylib             0x000000019cc0bdac _pthread_cond_wait + 1248
2   python.exe                          0x0000000100388394 PyCOND_TIMEDWAIT + 52 (condvar.h:73)
3   python.exe                          0x0000000100387584 take_gil + 180 (ceval_gil.c:380)
4   python.exe                          0x0000000100387ca0 PyEval_RestoreThread + 40 (ceval_gil.c:661)
5   syslog.cpython-312d-darwin.so       0x0000000101483334 syslog_syslog_impl + 160 (syslogmodule.c:212)
6   syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
7   python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
8   python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
9   python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
10  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
11  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
12  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
13  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
14  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
15  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
16  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
17  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
18  python.exe                          0x0000000100363d60 do_call_core + 532
19  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
20  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
21  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
22  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
23  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
24  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
25  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
26  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
27  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
28  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
29  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
30  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
31  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 8:
0   libsystem_kernel.dylib              0x000000019cbd4548 __psynch_cvwait + 8
1   libsystem_pthread.dylib             0x000000019cc0bdac _pthread_cond_wait + 1248
2   python.exe                          0x0000000100387968 drop_gil + 156 (ceval_gil.c:310)
3   python.exe                          0x0000000100388190 _Py_HandlePending + 168 (ceval_gil.c:972)
4   python.exe                          0x0000000100354bb4 _PyEval_EvalFrameDefault + 840 (ceval.c:1148)
5   python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
6   python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
7   python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
8   python.exe                          0x00000001003bf834 _PyObject_VectorcallTstate + 84 (pycore_call.h:92)
9   python.exe                          0x00000001003bf7d8 _PyObject_FastCallTstate + 12 (pycore_call.h:116)
10  python.exe                          0x00000001003bcd4c sys_audit_tstate + 532 (sysmodule.c:254)
11  python.exe                          0x00000001003bceac PySys_Audit + 52 (sysmodule.c:308)
12  syslog.cpython-312d-darwin.so       0x00000001014832c8 syslog_syslog_impl + 52 (syslogmodule.c:192)
13  syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
14  python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
15  python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
16  python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
17  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
18  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
19  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
20  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
21  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
22  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
23  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
24  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
25  python.exe                          0x0000000100363d60 do_call_core + 532
26  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
27  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
28  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
29  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
30  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
31  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
32  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
33  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
34  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
35  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
36  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
37  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
38  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 9 Crashed:
0   libsystem_kernel.dylib              0x000000019cbd8e68 __pthread_kill + 8
1   libsystem_pthread.dylib             0x000000019cc0b43c pthread_kill + 292
2   libsystem_c.dylib                   0x000000019caeae10 raise + 32
3   python.exe                          0x00000001003d49f0 faulthandler_fatal_error + 212 (faulthandler.c:372)
4   libsystem_platform.dylib            0x000000019cc56c44 _sigtramp + 56
5   ???                                 0xffff8001a14a5588 0 + 18446603343222166920
6   libsystem_asl.dylib                 0x00000001a14a5588 asl_msg_lookup + 76
7   libsystem_asl.dylib                 0x00000001a14a549c _asl_evaluate_send + 772
8   libsystem_asl.dylib                 0x00000001a14b7880 _vsyslog + 196
9   libsystem_asl.dylib                 0x00000001a14a900c syslog$DARWIN_EXTSN + 44
10  syslog.cpython-312d-darwin.so       0x000000010148332c syslog_syslog_impl + 152 (syslogmodule.c:211)
11  syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
12  python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
13  python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
14  python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
15  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
16  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
17  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
18  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
19  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
20  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
21  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
22  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
23  python.exe                          0x0000000100363d60 do_call_core + 532
24  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
25  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
26  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
27  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
28  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
29  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
30  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
31  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
32  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
33  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
34  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
35  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
36  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 10:                                                                                                                                                                                                                                                                                                                                                                                                                             0   libsystem_kernel.dylib              0x000000019cbd4548 __psynch_cvwait + 8                                                                                                                                                                                                                                                                                                                                                         1   libsystem_pthread.dylib             0x000000019cc0bdac _pthread_cond_wait + 1248                                                                                                                                                                                                                                                                                                                                                   2   python.exe                          0x0000000100388394 PyCOND_TIMEDWAIT + 52 (condvar.h:73)                                                                                                                                                                                                                                                                                                                                        3   python.exe                          0x0000000100387584 take_gil + 180 (ceval_gil.c:380)
4   python.exe                          0x0000000100388198 _Py_HandlePending + 176 (ceval_gil.c:976)
5   python.exe                          0x0000000100354bb4 _PyEval_EvalFrameDefault + 840 (ceval.c:1148)
6   python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
7   python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
8   python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
9   python.exe                          0x00000001003bf834 _PyObject_VectorcallTstate + 84 (pycore_call.h:92)
10  python.exe                          0x00000001003bf7d8 _PyObject_FastCallTstate + 12 (pycore_call.h:116)
11  python.exe                          0x00000001003bcd4c sys_audit_tstate + 532 (sysmodule.c:254)
12  python.exe                          0x00000001003bceac PySys_Audit + 52 (sysmodule.c:308)
13  syslog.cpython-312d-darwin.so       0x00000001014832c8 syslog_syslog_impl + 52 (syslogmodule.c:192)
14  syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
15  python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
16  python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
17  python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
18  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
19  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
20  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
21  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
22  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
23  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
24  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
25  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
26  python.exe                          0x0000000100363d60 do_call_core + 532
27  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
28  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
29  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
30  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
31  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
32  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
33  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
34  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
35  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
36  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
37  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
38  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
39  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 11:
0   libsystem_kernel.dylib              0x000000019cbd3a48 __psynch_mutexwait + 8
1   libsystem_pthread.dylib             0x000000019cc08918 _pthread_mutex_firstfit_lock_wait + 88
2   libsystem_pthread.dylib             0x000000019cc06244 _pthread_mutex_firstfit_lock_slow + 232
3   libsystem_pthread.dylib             0x000000019cc0be3c _pthread_cond_wait + 1392
4   python.exe                          0x0000000100388394 PyCOND_TIMEDWAIT + 52 (condvar.h:73)
5   python.exe                          0x0000000100387584 take_gil + 180 (ceval_gil.c:380)
6   python.exe                          0x0000000100387ca0 PyEval_RestoreThread + 40 (ceval_gil.c:661)
7   syslog.cpython-312d-darwin.so       0x0000000101483334 syslog_syslog_impl + 160 (syslogmodule.c:212)
8   syslog.cpython-312d-darwin.so       0x0000000101482dac syslog_syslog + 116 (syslogmodule.c.h:135)
9   python.exe                          0x00000001002d21e4 cfunction_call + 240 (methodobject.c:553)
10  python.exe                          0x000000010028cb28 _PyObject_MakeTpCall + 248 (call.c:215)
11  python.exe                          0x000000010028c87c _PyObject_VectorcallTstate + 176 (pycore_call.h:90)
12  python.exe                          0x000000010028d218 PyObject_Vectorcall + 56 (call.c:300)
13  python.exe                          0x000000010035e228 _PyEval_EvalFrameDefault + 39356 (ceval.c:4206)
14  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
15  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
16  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
17  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
18  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
19  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
20  python.exe                          0x0000000100363d60 do_call_core + 532
21  python.exe                          0x00000001003600f4 _PyEval_EvalFrameDefault + 47240 (ceval.c:4776)
22  python.exe                          0x00000001003547b0 _PyEval_EvalFrame + 36 (pycore_ceval.h:88)
23  python.exe                          0x00000001003546a8 _PyEval_Vector + 188 (ceval.c:5826)
24  python.exe                          0x000000010028d518 _PyFunction_Vectorcall + 108
25  python.exe                          0x000000010028fe4c _PyObject_VectorcallTstate + 116 (pycore_call.h:92)
26  python.exe                          0x000000010028f3ac method_vectorcall + 232 (classobject.c:67)
27  python.exe                          0x000000010028d1ac _PyVectorcall_Call + 236 (call.c:246)
28  python.exe                          0x000000010028d300 _PyObject_Call + 128 (call.c:329)
29  python.exe                          0x000000010028d3e0 PyObject_Call + 48 (call.c:356)
30  python.exe                          0x000000010041c380 thread_run + 80 (_threadmodule.c:1082)
31  python.exe                          0x00000001003c2284 pythread_wrapper + 28 (thread_pthread.h:230)
32  libsystem_pthread.dylib             0x000000019cc0b878 _pthread_start + 320
33  libsystem_pthread.dylib             0x000000019cc065e0 thread_start + 8

Thread 9 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x0000000000000000   x2: 0x0000000000000001   x3: 0x0000000000000000
    x4: 0x0000000000000000   x5: 0x0000000000000067   x6: 0x0000000000000000   x7: 0x00000000000008c0
    x8: 0xa3fa8558e12e537b   x9: 0xa3fa855999ebe37b  x10: 0x0000000000001bba  x11: 0x0000000213b63b5e
   x12: 0x0000000213b63b5e  x13: 0x0000000000000004  x14: 0x0000000000000881  x15: 0x000000008000001f
   x16: 0x0000000000000148  x17: 0x000000020b692e90  x18: 0x0000000000000000  x19: 0x000000000000000b
   x20: 0x0000000178c5b000  x21: 0x000000000000270f  x22: 0x0000000178c5b0e0  x23: 0x0000000000000000
   x24: 0x0000000208061000  x25: 0x0000000000070006  x26: 0x0000000208061000  x27: 0x0000000000000001
   x28: 0x00000001014a41f8   fp: 0x0000000178c5a080   lr: 0x000000019cc0b43c
    sp: 0x0000000178c5a060   pc: 0x000000019cbd8e68 cpsr: 0x40001000
   far: 0x00000001013ec020  esr: 0x56000080


Binary Images:
       0x10020c000 -        0x100517fff +python.exe (0) <F0B6BE46-AC61-39CB-854F-BCBE17E110D9> /Users/USER/*/python.exe
       0x100c18000 -        0x100c97fff  dyld (852.2) <38657979-1ABE-3C9A-BF64-EF3B746216AB> /usr/lib/dyld
       0x100f20000 -        0x100f23fff +termios.cpython-312d-darwin.so (0) <3DD6FC1C-36EF-38D5-B363-8F5212D5322B> /Users/USER/*/termios.cpython-312d-darwin.so
       0x100f30000 -        0x100f33fff +_heapq.cpython-312d-darwin.so (0) <9C9D0E1B-11DC-310D-B34C-D4A0CEF882C6> /Users/USER/*/_heapq.cpython-312d-darwin.so
       0x101040000 -        0x101043fff +_opcode.cpython-312d-darwin.so (0) <15926C6A-FC0A-3217-A0EC-1C329C79B3B1> /Users/USER/*/_opcode.cpython-312d-darwin.so
       0x101150000 -        0x10116bfff +_testcapi.cpython-312d-darwin.so (0) <4A8DB161-08FA-35EF-B87F-D01BD693A6AC> /Users/USER/*/_testcapi.cpython-312d-darwin.so
       0x101194000 -        0x1012a3fff +unicodedata.cpython-312d-darwin.so (0) <0A099185-7739-3C73-BE8A-DB8331F7B64A> /Users/USER/*/unicodedata.cpython-312d-darwin.so
       0x1012b4000 -        0x1012bffff +math.cpython-312d-darwin.so (0) <63FB8501-B25C-38FF-BD76-FF30E0062FCA> /Users/USER/*/math.cpython-312d-darwin.so
       0x1012d0000 -        0x1012d3fff +_bisect.cpython-312d-darwin.so (0) <4199D2F9-02FD-39DC-8719-F376DD3AC140> /Users/USER/*/_bisect.cpython-312d-darwin.so
       0x1012e0000 -        0x1012e3fff +_random.cpython-312d-darwin.so (0) <639FC4DE-8136-36D0-B6AD-1C064233C9CE> /Users/USER/*/_random.cpython-312d-darwin.so
       0x1012f0000 -        0x1012f3fff +_sha512.cpython-312d-darwin.so (0) <C5DA690D-4E67-37DF-86F9-4087F153726F> /Users/USER/*/_sha512.cpython-312d-darwin.so
       0x101300000 -        0x101307fff +zlib.cpython-312d-darwin.so (0) <9FB8F799-707E-37A1-A9CE-B0454A3162C9> /Users/USER/*/zlib.cpython-312d-darwin.so
       0x101318000 -        0x10131bfff +_bz2.cpython-312d-darwin.so (0) <174577CC-9AFD-3C64-A933-8617B4B81F2F> /Users/USER/*/_bz2.cpython-312d-darwin.so
       0x101328000 -        0x10132ffff +_lzma.cpython-312d-darwin.so (0) <A5CA474F-374C-30F4-828A-8DA280EE86B5> /Users/USER/*/_lzma.cpython-312d-darwin.so
       0x101340000 -        0x10135bfff +liblzma.5.dylib (0) <61710133-D7DB-31A2-B802-1048FF52D21C> /opt/homebrew/*/liblzma.5.dylib
       0x101470000 -        0x101473fff +resource.cpython-312d-darwin.so (0) <83B4F923-0490-38D7-BAF2-DDE9B171E7AF> /Users/USER/*/resource.cpython-312d-darwin.so
       0x101480000 -        0x101483fff +syslog.cpython-312d-darwin.so (0) <A1B4E866-52F2-3AC9-94DB-35AADC12AFEA> /Users/USER/*/syslog.cpython-312d-darwin.so
       0x19c966000 -        0x19c967fff  libsystem_blocks.dylib (79) <BA713F79-1620-38B2-8717-79849B8E897E> /usr/lib/system/libsystem_blocks.dylib
       0x19c968000 -        0x19c99ffff  libxpc.dylib (2038.120.1) <0E990C5D-C4F7-304A-9714-4409647CC2EF> /usr/lib/system/libxpc.dylib
       0x19c9a0000 -        0x19c9b7fff  libsystem_trace.dylib (1277.120.1) <7A6B4800-8D94-3091-AC78-2D6B97A9B496> /usr/lib/system/libsystem_trace.dylib
       0x19c9b8000 -        0x19ca2bfff  libcorecrypto.dylib (1000.140.4) <FAF9B49D-2583-34EF-850B-047F050ED996> /usr/lib/system/libcorecrypto.dylib
       0x19ca2c000 -        0x19ca57fff  libsystem_malloc.dylib (317.140.5) <A454F3F2-7BC4-3808-A60F-F5F4BC55577F> /usr/lib/system/libsystem_malloc.dylib
       0x19ca58000 -        0x19ca9cfff  libdispatch.dylib (1271.120.2) <4EDD5F72-2296-3891-B2A1-6741DB6C05C9> /usr/lib/system/libdispatch.dylib
       0x19ca9d000 -        0x19cad6fff  libobjc.A.dylib (824) <252C93CC-2D39-3C15-87F6-1336658B2F49> /usr/lib/libobjc.A.dylib
       0x19cad7000 -        0x19cad9fff  libsystem_featureflags.dylib (28.60.1) <61A79E0B-1836-376D-AC24-171B40A9FAE7> /usr/lib/system/libsystem_featureflags.dylib
       0x19cada000 -        0x19cb5bfff  libsystem_c.dylib (1439.141.1) <9B3C0AD0-1E0B-39D1-BE25-B97F8EA7E623> /usr/lib/system/libsystem_c.dylib
       0x19cb5c000 -        0x19cbb7fff  libc++.1.dylib (905.6) <BED05B96-CCAE-365A-B3F5-F8D72F5E77E1> /usr/lib/libc++.1.dylib
       0x19cbb8000 -        0x19cbcffff  libc++abi.dylib (905.6) <3C5176E2-CE48-312B-97F6-23D4E7AFA43F> /usr/lib/libc++abi.dylib
       0x19cbd0000 -        0x19cc03fff  libsystem_kernel.dylib (7195.141.6) <FA7E835C-CB30-3D98-9331-30CE6584423D> /usr/lib/system/libsystem_kernel.dylib
       0x19cc04000 -        0x19cc10fff  libsystem_pthread.dylib (454.120.2) <BDC1C5DA-9499-3580-9588-2928DE2440DD> /usr/lib/system/libsystem_pthread.dylib
       0x19cc11000 -        0x19cc52fff  libdyld.dylib (852.2) <BE0B539A-2EDF-3A99-9632-731C5928BBAC> /usr/lib/system/libdyld.dylib
       0x19cc53000 -        0x19cc59fff  libsystem_platform.dylib (254.80.2) <2CC11FAD-1E31-3533-B1D8-9CDDF5DFE595> /usr/lib/system/libsystem_platform.dylib
       0x19cc5a000 -        0x19cc85fff  libsystem_info.dylib (542.40.3) <3B5B9F97-ECFB-3F6E-9B85-D8C62A8B3205> /usr/lib/system/libsystem_info.dylib
       0x19cc86000 -        0x19d134fff  com.apple.CoreFoundation (6.9 - 1778.101) <1FC1BD60-DC83-3CC7-89AC-D734DC18473A> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
       0x19eda9000 -        0x19f01afff  libicucore.A.dylib (66112) <987F240B-946D-3F96-AD1A-F3EF6752D5DB> /usr/lib/libicucore.A.dylib
       0x19f01b000 -        0x19f025fff  libsystem_darwin.dylib (1439.141.1) <9BF54A49-874D-3D91-B319-6B2F676FE42D> /usr/lib/system/libsystem_darwin.dylib
       0x19f45e000 -        0x19f469fff  libsystem_notify.dylib (279.40.4) <FA85C044-99A5-352D-B7AD-45FB31B295FF> /usr/lib/system/libsystem_notify.dylib
       0x1a143d000 -        0x1a144cfff  libsystem_networkextension.dylib (1295.140.3) <0FFACDE5-D546-306D-B9FA-06EC8FCAEEF9> /usr/lib/system/libsystem_networkextension.dylib
       0x1a14a2000 -        0x1a14b9fff  libsystem_asl.dylib (385) <7F462B8A-541F-3195-9BAE-CDEDFA3BA8A4> /usr/lib/system/libsystem_asl.dylib
       0x1a2a13000 -        0x1a2a1afff  libsystem_symptoms.dylib (1431.140.1) <0B4FC817-CC87-31F2-872B-BD505D29F5EE> /usr/lib/system/libsystem_symptoms.dylib
       0x1a486f000 -        0x1a4880fff  libsystem_containermanager.dylib (318.100.4) <FCE2D8EA-A9D8-3A5D-A908-3E9C0AFFB941> /usr/lib/system/libsystem_containermanager.dylib
       0x1a54ed000 -        0x1a54f1fff  libsystem_configuration.dylib (1109.140.1) <9BC511C4-6613-3F1D-8592-7D3971B24921> /usr/lib/system/libsystem_configuration.dylib
       0x1a54f2000 -        0x1a54f6fff  libsystem_sandbox.dylib (1441.141.4) <CABF63A0-DDAF-3A54-8447-1256DFFD2907> /usr/lib/system/libsystem_sandbox.dylib
       0x1a5eb3000 -        0x1a5eb5fff  libquarantine.dylib (119.40.2) <DE9BFFD7-62ED-38ED-BF58-388F192BE6FC> /usr/lib/system/libquarantine.dylib
       0x1a6409000 -        0x1a6415fff  libbz2.1.0.dylib (44) <CAF8DA55-2A9E-3138-8177-62B54BF056D5> /usr/lib/libbz2.1.0.dylib
       0x1a6416000 -        0x1a641afff  libsystem_coreservices.dylib (127.1) <F107780A-EFAD-3DC8-A62F-825DD954128E> /usr/lib/system/libsystem_coreservices.dylib
       0x1a6600000 -        0x1a6611fff  libz.1.dylib (76) <4838F5D2-C9BB-366A-9745-209A80BCC426> /usr/lib/libz.1.dylib
       0x1a6612000 -        0x1a663ffff  libsystem_m.dylib (3186.100.3) <092A6565-3BE3-3EEF-844E-B846793A4115> /usr/lib/system/libsystem_m.dylib
       0x1a6641000 -        0x1a6646fff  libmacho.dylib (980) <F6CBC87D-DC83-3E5F-A7C0-B0115E7D1BC1> /usr/lib/system/libmacho.dylib
       0x1a665f000 -        0x1a666cfff  libcommonCrypto.dylib (60178.120.3) <146E6A52-3060-32A1-9119-C3D6F8B5C57B> /usr/lib/system/libcommonCrypto.dylib
       0x1a666d000 -        0x1a6677fff  libunwind.dylib (201) <2A01EF92-F92D-35F7-B638-8CBC6896DCAA> /usr/lib/system/libunwind.dylib
       0x1a6678000 -        0x1a667ffff  liboah.dylib (203.58) <6802DEC1-9ED5-3667-BFE0-3BDDE8BB40E1> /usr/lib/liboah.dylib
       0x1a6680000 -        0x1a6688fff  libcopyfile.dylib (173.40.2) <8477DD43-56AA-3422-BF23-4ADC7B98462F> /usr/lib/system/libcopyfile.dylib
       0x1a6689000 -        0x1a668cfff  libcompiler_rt.dylib (102.2) <50E8F23E-3E63-31BF-BFC3-EE75D281B151> /usr/lib/system/libcompiler_rt.dylib
       0x1a668d000 -        0x1a668ffff  libsystem_collections.dylib (1439.141.1) <EA89016C-2C72-3531-84D9-2A696F2DAC85> /usr/lib/system/libsystem_collections.dylib
       0x1a6690000 -        0x1a6692fff  libsystem_secinit.dylib (87.60.1) <836B23E3-DF08-31C8-98AD-4A9B12A4B404> /usr/lib/system/libsystem_secinit.dylib
       0x1a6693000 -        0x1a6695fff  libremovefile.dylib (49.120.1) <687A6E36-0460-30BC-A675-7F83059AE2D9> /usr/lib/system/libremovefile.dylib
       0x1a6696000 -        0x1a6696fff  libkeymgr.dylib (31) <C5D64EE9-AB76-3D4C-BC4F-2BE8415DC21A> /usr/lib/system/libkeymgr.dylib
       0x1a6697000 -        0x1a669ffff  libsystem_dnssd.dylib (1310.140.1) <DEE0783F-DDC9-35A6-9F6E-6B8FE94FBF0A> /usr/lib/system/libsystem_dnssd.dylib
       0x1a66a0000 -        0x1a66a5fff  libcache.dylib (83) <2EA812B3-0565-35FA-BF54-DF98EF3DB5DE> /usr/lib/system/libcache.dylib
       0x1a66a6000 -        0x1a66a7fff  libSystem.B.dylib (1292.120.1) <1E75FCDF-2357-30FE-AAAD-5290BA722464> /usr/lib/libSystem.B.dylib
       0x1a66a8000 -        0x1a66abfff  libfakelink.dylib (3) <F096DD26-38CB-3EC9-8FC1-2F8B11A0675D> /usr/lib/libfakelink.dylib
       0x1a66ac000 -        0x1a66acfff  com.apple.SoftLinking (1.0 - 1) <FA588B2E-82E9-3570-B6FE-D2AB5F2A3E28> /System/Library/PrivateFrameworks/SoftLinking.framework/Versions/A/SoftLinking
       0x1a9ac5000 -        0x1a9ac5fff  liblaunch.dylib (2038.120.1) <AA4E8477-1B82-357E-8275-FFED1F957AF9> /usr/lib/system/liblaunch.dylib
       0x1abf79000 -        0x1abf79fff  libsystem_product_info_filter.dylib (8.40.1) <14136F56-3690-3346-90A0-31FC34DF3E1B> /usr/lib/system/libsystem_product_info_filter.dylib

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=509.8M resident=0K(0%) swapped_out_or_unallocated=509.8M(100%)
Writable regions: Total=489.4M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=489.4M(100%)

                                VIRTUAL   REGION
REGION TYPE                        SIZE    COUNT (non-coalesced)
===========                     =======  =======
Activity Tracing                   256K        1
Kernel Alloc Once                   32K        1
MALLOC                            49.1M       20
MALLOC guard page                   96K        5
MALLOC_MEDIUM (reserved)         240.0M        2         reserved VM address space (unallocated)
STACK GUARD                        192K       12
Stack                            192.3M       12
VM_ALLOCATE                        1.0G       12
__AUTH                              53K       12
__AUTH_CONST                      1033K       44
__DATA                            2089K       57
__DATA_CONST                      2741K       58
__DATA_DIRTY                       110K       23
__LINKEDIT                       493.4M       19
__OBJC_CONST                        37K        6
__OBJC_RO                         75.1M        1
__OBJC_RW                         2576K        1
__TEXT                            16.4M       65
__UNICODE                          588K        1
shared memory                       32K        2
===========                     =======  =======
TOTAL                              2.1G      354
TOTAL, minus reserved VM space     1.8G      354

@pablogsal
Copy link
Member

Ok, after activating developer mode I can lldb:

* thread #7, stop reason = EXC_BAD_ACCESS (code=1, address=0xffffffffffffffff)
  * frame #0: 0x00000001a14a4190 libsystem_asl.dylib`_asl_msg_index + 60
    frame #1: 0x00000001a14a5588 libsystem_asl.dylib`asl_msg_lookup + 76
    frame #2: 0x00000001a14a549c libsystem_asl.dylib`_asl_evaluate_send + 772
    frame #3: 0x00000001a14b7880 libsystem_asl.dylib`_vsyslog + 196
    frame #4: 0x00000001a14a900c libsystem_asl.dylib`syslog$DARWIN_EXTSN + 44
    frame #5: 0x0000000101e3332c syslog.cpython-312d-darwin.so`syslog_syslog_impl(module=<unavailable>, group_left_1=<unavailable>, priority=6, message="test message from python test_syslog") at syslogmodule.c:211:5 [opt]
    frame #6: 0x0000000101e32dac syslog.cpython-312d-darwin.so`syslog_syslog(module=0x0000000101da6c30, args=0x0000000101d9c7d0) at syslogmodule.c.h:135:20 [opt]
    frame #7: 0x00000001000c61e4 python.exe`cfunction_call(func=0x0000000101da6cf0, args=0x0000000101d9c7d0, kwargs=0x0000000000000000) at methodobject.c:553:18 [opt]
    frame #8: 0x0000000100080b28 python.exe`_PyObject_MakeTpCall(tstate=0x0000000100e049f0, callable=0x0000000101da6cf0, args=0x0000000101e8c1f0, nargs=<unavailable>, keywords=0x0000000000000000) at call.c:215:18 [opt]
    frame #9: 0x000000010008087c python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100e049f0, callable=0x0000000101da6cf0, args=0x0000000101e8c1f0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at pycore_call.h:90:16 [opt]
    frame #10: 0x0000000100081218 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:300:12 [opt]
    frame #11: 0x0000000100152228 python.exe`_PyEval_EvalFrameDefault(tstate=<unavailable>, frame=0x0000000101e8c188, throwflag=0) at ceval.c:4206:23 [opt]
    frame #12: 0x00000001001487b0 python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:88:16 [opt]
    frame #13: 0x00000001001486a8 python.exe`_PyEval_Vector(tstate=0x0000000100e049f0, func=0x0000000100b81e90, locals=0x0000000000000000, args=0x000000010043fef0, argcount=0, kwnames=<unavailable>) at ceval.c:5826:24 [opt]
    frame #14: 0x0000000100081518 python.exe`_PyFunction_Vectorcall(func=0x0000000100b81e90, stack=0x000000010043fef0, nargsf=<unavailable>, kwnames=0x0000000000000000) at call.c:0:45 [opt]
    frame #15: 0x00000001000811ac python.exe`_PyVectorcall_Call(tstate=0x0000000100e049f0, func=(python.exe`_PyFunction_Vectorcall at call.c:387), callable=0x0000000100b81e90, tuple=0x000000010043fed8, kwargs=0x0000000101da3a10) at call.c:246:16 [opt]
    frame #16: 0x0000000100081300 python.exe`_PyObject_Call(tstate=0x0000000100e049f0, callable=0x0000000100b81e90, args=0x000000010043fed8, kwargs=0x0000000101da3a10) at call.c:329:16 [opt]
    frame #17: 0x00000001000813e0 python.exe`PyObject_Call(callable=<unavailable>, args=<unavailable>, kwargs=<unavailable>) at call.c:356:12 [opt]
    frame #18: 0x0000000100157d60 python.exe`do_call_core(tstate=0x0000000100e049f0, func=0x0000000100b81e90, callargs=0x000000010043fed8, kwdict=0x0000000101da3a10, use_tracing=<unavailable>) at ceval.c:0:13 [opt]
    frame #19: 0x00000001001540f4 python.exe`_PyEval_EvalFrameDefault(tstate=<unavailable>, frame=0x0000000101e8c110, throwflag=0) at ceval.c:4776:22 [opt]
    frame #20: 0x00000001001487b0 python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:88:16 [opt]
    frame #21: 0x00000001001486a8 python.exe`_PyEval_Vector(tstate=0x0000000100e049f0, func=0x0000000101d68ec0, locals=0x0000000000000000, args=0x0000000175e42e38, argcount=1, kwnames=<unavailable>) at ceval.c:5826:24 [opt]
    frame #22: 0x0000000100081518 python.exe`_PyFunction_Vectorcall(func=0x0000000101d68ec0, stack=0x0000000175e42e38, nargsf=<unavailable>, kwnames=0x0000000000000000) at call.c:0:45 [opt]
    frame #23: 0x0000000100083e4c python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100e049f0, callable=0x0000000101d68ec0, args=0x0000000175e42e38, nargsf=1, kwnames=0x0000000000000000) at pycore_call.h:92:11 [opt]
    frame #24: 0x00000001000833ac python.exe`method_vectorcall(method=<unavailable>, args=<unavailable>, nargsf=0, kwnames=0x0000000000000000) at classobject.c:67:20 [opt]
    frame #25: 0x00000001000811ac python.exe`_PyVectorcall_Call(tstate=0x0000000100e049f0, func=(python.exe`method_vectorcall at classobject.c:44), callable=0x0000000101da3950, tuple=0x000000010043fed8, kwargs=0x0000000000000000) at call.c:246:16 [opt]
    frame #26: 0x0000000100081300 python.exe`_PyObject_Call(tstate=0x0000000100e049f0, callable=0x0000000101da3950, args=0x000000010043fed8, kwargs=0x0000000000000000) at call.c:329:16 [opt]
    frame #27: 0x00000001000813e0 python.exe`PyObject_Call(callable=<unavailable>, args=<unavailable>, kwargs=<unavailable>) at call.c:356:12 [opt]
    frame #28: 0x0000000100210380 python.exe`thread_run(boot_raw=0x0000000101d9c040) at _threadmodule.c:1082:21 [opt]
    frame #29: 0x00000001001b6284 python.exe`pythread_wrapper(arg=<unavailable>) at thread_pthread.h:230:5 [opt]
    frame #30: 0x000000019cc0b878 libsystem_pthread.dylib`_pthread_start + 320

Here are the frames before entering the libsystem_asl.dlyb calls:

libsystem_asl.dylib`vsyslog:
    0x1a14a9018 <+0>:  mov    x3, x30
(lldb)
syslog.cpython-312d-darwin.so was compiled with optimization - stepping may behave oddly; variables may not be available.
frame #5: 0x0000000101e3332c syslog.cpython-312d-darwin.so`syslog_syslog_impl(module=<unavailable>, group_left_1=<unavailable>, priority=6, message="test message from python test_syslog") at syslogmodule.c:211:5 [opt]
   208      PyObject *ident = S_ident_o;
   209      Py_XINCREF(ident);
   210      Py_BEGIN_ALLOW_THREADS;
-> 211      syslog(priority, "%s", message);
   212      Py_END_ALLOW_THREADS;
   213      Py_XDECREF(ident);
   214      Py_RETURN_NONE;
(lldb)
frame #6: 0x0000000101e32dac syslog.cpython-312d-darwin.so`syslog_syslog(module=0x0000000101da6c30, args=0x0000000101d9c7d0) at syslogmodule.c.h:135:20 [opt]
   132              PyErr_SetString(PyExc_TypeError, "syslog.syslog requires 1 to 2 arguments");
   133              goto exit;
   134      }
-> 135      return_value = syslog_syslog_impl(module, group_left_1, priority, message);
   136
   137  exit:
   138      return return_value;
(lldb)
python.exe was compiled with optimization - stepping may behave oddly; variables may not be available.
frame #7: 0x00000001000c61e4 python.exe`cfunction_call(func=0x0000000101da6cf0, args=0x0000000101d9c7d0, kwargs=0x0000000000000000) at methodobject.c:553:18 [opt]
   550                            ((PyCFunctionObject*)func)->m_ml->ml_name);
   551              return NULL;
   552          }
-> 553          result = _PyCFunction_TrampolineCall(meth, self, args);
   554      }
   555      return _Py_CheckFunctionResult(tstate, func, result, NULL);
   556  }
(lldb)
frame #8: 0x0000000100080b28 python.exe`_PyObject_MakeTpCall(tstate=0x0000000100e049f0, callable=0x0000000101da6cf0, args=0x0000000101e8c1f0, nargs=<unavailable>, keywords=0x0000000000000000) at call.c:215:18 [opt]
   212      PyObject *result = NULL;
   213      if (_Py_EnterRecursiveCallTstate(tstate, " while calling a Python object") == 0)
   214      {
-> 215          result = _PyCFunctionWithKeywords_TrampolineCall(
   216              (PyCFunctionWithKeywords)call, callable, argstuple, kwdict);
   217          _Py_LeaveRecursiveCallTstate(tstate);
   218      }
(lldb)
frame #9: 0x000000010008087c python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100e049f0, callable=0x0000000101da6cf0, args=0x0000000101e8c1f0, nargsf=9223372036854775809, kwnames=0x0000000000000000) at pycore_call.h:90:16 [opt]
   87       func = _PyVectorcall_FunctionInline(callable);
   88       if (func == NULL) {
   89           Py_ssize_t nargs = PyVectorcall_NARGS(nargsf);
-> 90           return _PyObject_MakeTpCall(tstate, callable, args, nargs, kwnames);
   91       }
   92       res = func(callable, args, nargsf, kwnames);
   93       return _Py_CheckFunctionResult(tstate, callable, res, NULL);
(lldb)
frame #10: 0x0000000100081218 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:300:12 [opt]
   297                       size_t nargsf, PyObject *kwnames)
   298  {
   299      PyThreadState *tstate = _PyThreadState_GET();
-> 300      return _PyObject_VectorcallTstate(tstate, callable,
   301                                        args, nargsf, kwnames);
   302  }

@pablogsal
Copy link
Member

I can confirm that calling the function with the GIL held fixes the problem.

@ronaldoussoren
Copy link
Contributor

Calling with the GIL held fixes the crash for concurrent calls from Python, but that's the best we can do.

I want to try to create a simple reproducer in C and file a issue with Apple about this (not that I expect this to have any effect :-( ). With some luck this might also tell us if newer versions of the OS fix the issue, which would allow us to drop the workaround later on.

miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 13, 2022
On macOS, fix a crash in syslog.syslog() in multi-threaded
applications. On macOS, the libc syslog() function is not
thread-safe, so syslog.syslog() no longer releases the GIL to call
it.
(cherry picked from commit d4b9166)

Co-authored-by: Victor Stinner <[email protected]>
ambv pushed a commit that referenced this issue Oct 13, 2022
On macOS, fix a crash in syslog.syslog() in multi-threaded
applications. On macOS, the libc syslog() function is not
thread-safe, so syslog.syslog() no longer releases the GIL to call
it.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Oct 13, 2022
On macOS, fix a crash in syslog.syslog() in multi-threaded
applications. On macOS, the libc syslog() function is not
thread-safe, so syslog.syslog() no longer releases the GIL to call
it.
(cherry picked from commit d4b9166)

Co-authored-by: Victor Stinner <[email protected]>
@vstinner
Copy link
Member Author

Well, for me it's quite clear the macOS syslog implementation is not thread-safe, and so using a lock (like the GIL) in Python is the right solution to make the code correct.

* thread #7, stop reason = EXC_BAD_ACCESS (code=1, address=0xffffffffffffffff)
  * frame #0: 0x00000001a14a4190 libsystem_asl.dylib`_asl_msg_index + 60
    frame #1: 0x00000001a14a5588 libsystem_asl.dylib`asl_msg_lookup + 76
    frame #2: 0x00000001a14a549c libsystem_asl.dylib`_asl_evaluate_send + 772
    frame #3: 0x00000001a14b7880 libsystem_asl.dylib`_vsyslog + 196
    frame #4: 0x00000001a14a900c libsystem_asl.dylib`syslog$DARWIN_EXTSN + 44

The source code can be found in Apple syslog source code:

But frame locall variables are not included in the trace, and I don't know how to get the C line numbre from the trace (_asl_msg_index + 60). I don't have access to macOS.

_asl_evaluate_send() uses a lock, but later, not at the beginning of the function which calls asl_msg_lookup().

I cannot find anything like a mutex or a lock in asl_msg.c.


2019, Someone else hitting a similar syslog() crash on macOS with threads: rigetti/rpcq#92 (comment)

man 3 asl on my macbook says: The state information in a client handle is not protected by locking or thread synchronization mechanisms, except for one special case where NULL is used as a client handle. That special case is described below. (...)

asl manual page: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/asl.3.html

When logging from multiple threads, each thread must open a separate client handle using asl_open.

@vstinner
Copy link
Member Author

The Python syslog.syslog() function calls openlog() once, and then store the result. So all threads use the same log "handle".

/*  only one instance, only one syslog, so globals should be ok  */                
static PyObject *S_ident_o = NULL;                      /*  identifier, held by openlog()  */
static char S_log_open = 0;                                                        

The syslog C API doesn't accept any "log handle": it's stored somewhere by the C library between openlog() and syslog() calls.

The unit test is a stress test for multi-threading: it calls Python syslog.openlog() and syslog.syslog() functions in a loop from many threads in parallel. Since the GIL is released, magic happens :-)

            threads = [threading.Thread(target=opener)]
            threads += [threading.Thread(target=logger) for k in range(10)]
            with threading_helper.start_threads(threads):
                start.set()
                time.sleep(0.1)
                stop = True

It seems like these threads only work for 0.1 second. If you want to make the crash more likely, replace sleep(0.1) with sleep(30).

Maybe sometimes, if openlog() and syslog() are called at the same time from two threads, there is a race condition in syslog() and there is a crash.

It's a little bit weird that openlog() and syslog() don't have any kind of locking since the API is somehow "state less": no "state" is passed to syslog(). The internal asl handle is stored internally by openlog().

miss-islington added a commit that referenced this issue Oct 13, 2022
On macOS, fix a crash in syslog.syslog() in multi-threaded
applications. On macOS, the libc syslog() function is not
thread-safe, so syslog.syslog() no longer releases the GIL to call
it.
(cherry picked from commit d4b9166)

Co-authored-by: Victor Stinner <[email protected]>
miss-islington added a commit that referenced this issue Oct 13, 2022
On macOS, fix a crash in syslog.syslog() in multi-threaded
applications. On macOS, the libc syslog() function is not
thread-safe, so syslog.syslog() no longer releases the GIL to call
it.
(cherry picked from commit d4b9166)

Co-authored-by: Victor Stinner <[email protected]>
@ronaldoussoren
Copy link
Contributor

Looking at the code I linked to earlier the problem is a race condition between calls to openlog and syslog inside libc.

  • openlog releases the ASL handle (if any) and creates a new one, this path is guarded by a lock
  • syslog uses the ASL handle without locking and hence might grab the handle in a call to an ASL API just before it is reset after which things go boom when ASL tries to use the handle.

What I don't understand at all is why someone took the effort of adding a mutex to guard changing the handle in openlog, but didn't add a similar guard when using the handle later on. In normal code your very unlikely to hit this race condition because applications generally only call openlog once, but still.

@vstinner
Copy link
Member Author

My PR #98238 got merged, so I close the issue.

If someone wants to fix/enhance macOS, I suggest you opening a bug report on the macOS side instead ;-)

@ambv
Copy link
Contributor

ambv commented Oct 13, 2022

It seems like these threads only work for 0.1 second. If you want to make the crash more likely, replace sleep(0.1) with sleep(30).

I tried that on macOS 12.6 and couldn't reproduce with -F -j6 when running for over an hour:

1:17:52 load avg: 61.52 [868] test_syslog passed (30.8 sec)

I even tried to set the sleep to 0.1 * randint(1, 100) and -j20 to force more concurrent syslog/openlog calls but that didn't reproduce either when running for an hour:

1:34:06 load avg: 150.95 [14513] test_syslog passed

Next I'll try my old Intel laptop with macOS 10.15.

@ronaldoussoren
Copy link
Contributor

It looks like the issue is fixed in macOS 12, the latest sources are on GitHub instead of opensource.apple.com and manage the ASL handle more sanely: https://github.com/apple-oss-distributions/syslog/blob/syslog-392.100.2/libsystem_asl.tproj/src/syslog.c

In this version the ASL handle used in syslog() is a new reference instead of a borrowed reference (to use Python terminology). That explains why we couldn't reproduce the issue on macOS 12 systems.

@vstinner
Copy link
Member Author

In that case, the Python syslog module should hold or not the GIL depending on the macOS version.


On macOS, the Python select module checks at runtime if poll() is broken or not, to workaround another macOS issue:

/*
 * On some systems poll() sets errno on invalid file descriptors. We test
 * for this at runtime because this bug may be fixed or introduced between
 * OS releases.
 */
static int select_have_broken_poll(void) { ... }

carljm added a commit to carljm/cpython that referenced this issue Oct 14, 2022
* main: (38 commits)
  pythongh-98251: Allow venv to pass along PYTHON* variables to pip and ensurepip when they do not impact path resolution (pythonGH-98259)
  Bpo-41246: IOCP Proactor avoid callback code duplication (python#21399)
  bpo-46364: Use sockets for stdin of asyncio only on AIX (python#30596)
  pythongh-98178: syslog() is not thread-safe on macOS (python#98213)
  Mark all targets in `Doc/Makefile` as `PHONY` (pythonGH-98189)
  pythongh-97982: Factorize PyUnicode_Count() and unicode_count() code (python#98025)
  pythongh-96265: Formatting changes for faq/general (python#98129)
  tutorial: remove "with single quotes" (python#98204)
  pythongh-97669: Remove Tools/scripts/startuptime.py (python#98214)
  signalmodule.c uses _PyErr_WriteUnraisableMsg() (python#98217)
  pythongh-97669: Fix test_tools reference leak (python#98216)
  pythongh-97669: Create Tools/patchcheck/ directory (python#98186)
  pythongh-65046: Link to logging cookbook from asyncio docs (python#98207)
  Formatting fixes in contextlib docs (python#98111)
  pythongh-95276: Add callable entry to the glossary (python#95738)
  pythongh-96130: Rephrase use of "typecheck" verb for clarity (python#98144)
  Fix some incorrect indentation around the main switch (python#98177)
  pythongh-98172: Fix formatting in `except*` docs (python#98173)
  pythongh-97982: Remove asciilib_count() (python#98164)
  pythongh-95756: Free and NULL-out code caches when needed (pythonGH-98181)
  ...
carljm added a commit to carljm/cpython that referenced this issue Oct 14, 2022
* main: (37 commits)
  pythongh-98251: Allow venv to pass along PYTHON* variables to pip and ensurepip when they do not impact path resolution (pythonGH-98259)
  Bpo-41246: IOCP Proactor avoid callback code duplication (python#21399)
  bpo-46364: Use sockets for stdin of asyncio only on AIX (python#30596)
  pythongh-98178: syslog() is not thread-safe on macOS (python#98213)
  Mark all targets in `Doc/Makefile` as `PHONY` (pythonGH-98189)
  pythongh-97982: Factorize PyUnicode_Count() and unicode_count() code (python#98025)
  pythongh-96265: Formatting changes for faq/general (python#98129)
  tutorial: remove "with single quotes" (python#98204)
  pythongh-97669: Remove Tools/scripts/startuptime.py (python#98214)
  signalmodule.c uses _PyErr_WriteUnraisableMsg() (python#98217)
  pythongh-97669: Fix test_tools reference leak (python#98216)
  pythongh-97669: Create Tools/patchcheck/ directory (python#98186)
  pythongh-65046: Link to logging cookbook from asyncio docs (python#98207)
  Formatting fixes in contextlib docs (python#98111)
  pythongh-95276: Add callable entry to the glossary (python#95738)
  pythongh-96130: Rephrase use of "typecheck" verb for clarity (python#98144)
  Fix some incorrect indentation around the main switch (python#98177)
  pythongh-98172: Fix formatting in `except*` docs (python#98173)
  pythongh-97982: Remove asciilib_count() (python#98164)
  pythongh-95756: Free and NULL-out code caches when needed (pythonGH-98181)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

8 participants