Undeclared function 'is_pad' on macos-13 GitHub runners

[colesbury]

Bug report

It looks like it's only on the macos-13 runner. I'm not sure what changed. The failures look to be on unrelated commits and PRs.

./Modules/_cursesmodule.c:1336:9: error: call to undeclared function 'is_pad'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    if (py_is_pad(self->win)) {
        ^
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^
./Modules/_cursesmodule.c:2024:9: error: call to undeclared function 'is_pad'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    if (py_is_pad(self->win)) {
        ^
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^
./Modules/_cursesmodule.c:2248:9: error: call to undeclared function 'is_pad'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    if (py_is_pad(self->win)) {
        ^
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^
./Modules/_cursesmodule.c:2324:9: error: call to undeclared function 'is_pad'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    if (py_is_pad(self->win)) {
        ^
./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
#define py_is_pad(win)      is_pad(win)
                            ^
4 errors generated.
make: *** [Modules/_cursesmodule.o] Error 1

Seen in:

• main branch ea25f32: https://github.com/python/cpython/actions/runs/7879856175/job/21500903793
• PR 115190: https://github.com/python/cpython/actions/runs/7879468846/job/21499789192?pr=115190

Linked PRs

• gh-115383: Install ncurses in reusable macos workflow #115389
• gh-115383: Use runner version to compute config.cache key #115409
• [3.12] gh-115383: Use runner version to compute config.cache key (GH-115409) #115427
• [3.11] gh-115383: Use runner version to compute config.cache key (GH-115409) #115428

[hugovk]

Theory: those failures are both using version 20240114.1 of the runner image, passing builds are using newer 20240204.1 (visible in the logs under Set up job > Runner Image).

Perhaps some tools/compilers have changed and we're restoring some caches for the wrong ones?

• https://github.com/actions/runner-images/blob/macos-13/20240114.1/images/macos/macos-13-Readme.md
• https://github.com/actions/runner-images/blob/macos-13/20240204.1/images/macos/macos-13-Readme.md

[vstinner]

@sobolevn: oh, do you recall that issue?

[sobolevn]

Docs: https://manpages.debian.org/experimental/ncurses-doc/is_pad.3ncurses.en.html
We have this configuration:

{ printf "%s\n" "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_curses_is_pad" >&5
printf "%s\n" "$ac_cv_lib_curses_is_pad" >&6; }
  if test "x$ac_cv_lib_curses_is_pad" = xyes
then :

printf "%s\n" "#define HAVE_CURSES_IS_PAD 1" >>confdefs.h

fi

And then we use this macro:

cpython/Modules/_cursesmodule.c

Lines 1157 to 1163 in 0a6e1a4

	#if defined(HAVE_CURSES_IS_PAD)
	#define py_is_pad(win) is_pad(win)
	#elif defined(WINDOW_HAS_FLAGS)
	#define py_is_pad(win) ((win) ? ((win)->_flags & _ISPAD) != 0 : FALSE)
	#endif

Refs 8bc7d63

Interesting that the configure did detect is_pad:

checking for curses function is_pad... (cached) yes

I think that the only thing that can change here is the curses version.

[sobolevn]

Let's try to brew install ncurses?

[sobolevn]

Ok, it didn't work. The second idea: we can try to delete the builder cache: https://github.com/python/cpython/actions/caches?query=macos (@hugovk?)

[hugovk]

We could try that, but it might re-occur as new caches are saved. It could at least help isolate the problem.

If this is only a problem with the older 20240114.1 images, I expect they're currently rolling out the newer 20240204.1 across the builders, so the problem would eventually go away. But it could potentially re-occur. Perhaps we also need to add an extra key to the cache.

---

Comparing failing/passing logs:

The failure is with Xcode 14.3.1:

2024-02-13T01:03:53.7711130Z ./Modules/_cursesmodule.c:1336:9: error: call to undeclared function 'is_pad'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
2024-02-13T01:03:53.8018530Z     if (py_is_pad(self->win)) {
2024-02-13T01:03:53.8019670Z         ^
2024-02-13T01:03:53.8021030Z ./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
2024-02-13T01:03:53.8022380Z #define py_is_pad(win)      is_pad(win)
2024-02-13T01:03:53.8023430Z                             ^

The passing build is with Xcode 15.0.1, we don't get an error, but we do get a warning:

2024-02-13T06:33:40.8044080Z ./Modules/_cursesmodule.c:1336:9: warning: 'is_pad' is only available on macOS 14.0 or newer [-Wunguarded-availability-new]
2024-02-13T06:33:40.8160730Z     if (py_is_pad(self->win)) {
2024-02-13T06:33:40.8185070Z         ^~~~~~~~~~~~~~~~~~~~
2024-02-13T06:33:40.8336470Z ./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
2024-02-13T06:33:40.8340060Z gcc -I./Modules/_sqlite -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal -I./Include/internal/mimalloc  -I. -I./Include     -c ./Modules/_sqlite/connection.c -o Modules/_sqlite/connection.o
2024-02-13T06:33:40.8362630Z #define py_is_pad(win)      is_pad(win)
2024-02-13T06:33:40.8366430Z                             ^~~~~~
2024-02-13T06:33:40.8389540Z /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/ncurses.h:964:29: note: 'is_pad' has been marked as being introduced in macOS 14.0 here, but the deployment target is macOS 13.6.0
2024-02-13T06:33:40.8392810Z extern NCURSES_EXPORT(bool) is_pad (const WINDOW *);            /* generated */
2024-02-13T06:33:40.8501350Z                             ^
2024-02-13T06:33:40.8527550Z ./Modules/_cursesmodule.c:1336:9: note: enclose 'is_pad' in a __builtin_available check to silence this warning
2024-02-13T06:33:40.8590660Z     if (py_is_pad(self->win)) {
2024-02-13T06:33:40.8773050Z         ^~~~~~~~~~~~~~~~~~~~
2024-02-13T06:33:40.9074800Z ./Modules/_cursesmodule.c:1159:29: note: expanded from macro 'py_is_pad'
2024-02-13T06:33:40.9322140Z #define py_is_pad(win)      is_pad(win)
2024-02-13T06:33:40.9325580Z gcc -I./Modules/_sqlite -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal -I./Include/internal/mimalloc  -I. -I./Include     -c ./Modules/_sqlite/cursor.c -o Modules/_sqlite/cursor.o
2024-02-13T06:33:40.9348610Z                             ^~~~~~

[hugovk]

• actions/runner-images@macos-13/20240114.1/images/macos/macos-13-Readme.md
• actions/runner-images@macos-13/20240204.1/images/macos/macos-13-Readme.md

Partial diff of these:

[hugovk]

Related? #109617 "Refresh Screen Provided By curses.wrapper Causes Seg Fault (macOS, xcode 15 Apple supplied ncurses 6.0 breakage)"

[sobolevn]

This looks like a fix #111258

[hugovk]

That looks good: #111258 (comment)

[sorcio]

Note that #111258 applies only to builds that use Apple-provided ncurses. If I remember correctly, official macOS builds ship a specific ncurses and are not affected by either issue. So this is one way that GitHub runners are not aligned with official builds, in a way that is not well documented.

[colesbury]

Do you think that there is also a configure caching bug? The passing builds that @hugovk linked to did not restore config.cache, while the failing builds restored it.

[hugovk]

Comparing failing/passing logs:

These both restored the same cache:

Cache Size: ~0 MB (7683 B)
/usr/local/bin/gtar -xf /Users/runner/work/_temp/a2443c9c-6079-42bf-b7e1-fb95aae8bcc8/cache.tzst -P -C /Users/runner/work/cpython/cpython --delay-directory-restore --use-compress-program unzstd
Cache restored successfully
Cache restored from key: build_macos-macos-13-f1718b5f1cadc1e4c790771a233a0dec093844a30a6c2225822d97c5c78a6cbd

Both macos-13, but different Xcode versions. It sounds plausible a cache from one Xcode version is problematic when restored for another Xcode version?

Caching the image version like in #115409 sounds promising!

[hugovk]

From #111258 (comment):

The free-threaded build ran on the older 20240114.1 with Xcode 14 (which has recently started failing):

• python/cpython/actions/runs/7883963392/job/21511964656?pr=111258

And the regular build ran on the newer 20240204.1 with Xcode 15:

• python/cpython/actions/runs/7883963392/job/21511964137?pr=111258

And both passed 👍

Both ran with no cache restored:

Cache not found for input keys: build_macos-macos-13-1fbf1eb725ce1b5db0f36e7494693622bbfb08954e2caf20404e308fed7ff9f9

[colesbury]

These both restored the same cache...

Yeah, my comment wasn't accurate. I should have said the failing builds appear to be both running image 20240114.1 AND also restore config.cache. For example, here's a job that passes: even though it uses 20240114.1, it doesn't restore from cache (because it changes configure.ac) https://github.com/python/cpython/actions/runs/7888270010/job/21525416714.

[itamaro]

anything left to do here?

[hugovk]

CI is green, let's close. Thanks everyone for the investigation!

[ambv]

We're hitting this again with the pyrepl PR. The image now is at 20240421.1 so way past the versions discussed above. I'm leaning towards landing GH-111258 to address this properly.

[ambv]

The PR landed and fixed the issue on our PR, too. This can be closed again :D