Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Builds outside source folder broken, failed import of _importlib - no detection or valid error message or recommended cleanup. #112305

Closed
nascheme opened this issue Nov 21, 2023 · 8 comments
Assignees
Labels
3.11 only security fixes 3.12 bugs and security fixes 3.13 bugs and security fixes build The build process and cross-build type-bug An unexpected behavior, bug, or error

Comments

@nascheme
Copy link
Member

nascheme commented Nov 21, 2023

Bug description:

This bug was exposed by GH-108716. I think it was not the cause, just that it exposed an existing issue.

The build fails with:

./python -E -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform
Fatal Python error: _PyImport_InitCore: failed to initialize importlib
Python runtime state: preinitialized
ImportError: Frozen object named '_frozen_importlib' is invalid

Current thread 0x00007fb629644740 (most recent call first):
  <no Python frame>
make: *** [Makefile:932: platform] Error 1

Likely python doesn't know to look in $(srcdir)/Lib for the _importlib library. It probably looks in ./Lib/ and that fails.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

@nascheme nascheme added the type-bug An unexpected behavior, bug, or error label Nov 21, 2023
@AlexWaygood AlexWaygood added the build The build process and cross-build label Nov 21, 2023
@gpshead
Copy link
Member

gpshead commented Dec 19, 2023

I build out of tree all the time on Linux without trouble. But this hits me every time on macOS these days.

I am often (not always) unable to build on a Mac in the best way to do all development: out of source tree builds.

b$ ./python.exe -E -S
Fatal Python error: _PyImport_InitCore: failed to initialize importlib
Python runtime state: preinitialized
Exception ignored in the internal traceback machinery:
SystemError: null argument to internal routine
ImportError: Frozen object named '_frozen_importlib' is invalid

the build step hitting that is the generate sysconfig data one.

@gpshead
Copy link
Member

gpshead commented Dec 19, 2023

I managed to "fix" it on one system by doing an in-tree build, followed by a make distclean. now my out of tree builds work.

this smells like something not-quite up to date and synchronized related to _frozen_importlib when taken from a git checkout maybe?

@erlend-aasland
Copy link
Contributor

Out-of-tree builds require a clean source tree; I recommend doing a git clean -fdx before starting the out-of-tree build.

@nascheme
Copy link
Member Author

I tested with a clean checkout and a build in a sub-folder and it works. Closing.

@gpshead
Copy link
Member

gpshead commented Dec 20, 2023

We do still have a problem here: My git repos often wind up in this state and I never do in-tree builds and nothing in our build process detects whatever the underlying issue is and provides any form of cleanup. So winding up in Fatal Python error: _PyImport_InitCore: failed to initialize importlib will continue to happen to people.

I don't want our answer to this to be "give up, delete your repo, and start over".

git clean -fdx did not solve my problem.

@gpshead gpshead reopened this Dec 20, 2023
@gpshead gpshead changed the title Builds outside source folder broken, failed import of _importlib Builds outside source folder broken, failed import of _importlib - no detection or valid error message or recommended cleanup. Dec 20, 2023
@gpshead
Copy link
Member

gpshead commented Dec 20, 2023

I'm leaving this open because so long as i've got a tree on one machine that leads to this, i've at least got a datapoint where I can investigate further what is going on.

my workflow involves frequently switching branches in that git repo, including old stable release branches to modern branches. I don't believe we always do the right thing in this back and forth situation. some dirtiness within the source tree from our runtime or build process that is ignored by git seems to accumulate.

@gpshead gpshead self-assigned this Dec 21, 2023
gpshead added a commit to gpshead/cpython that referenced this issue Dec 21, 2023
A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
@gpshead gpshead added 3.11 only security fixes 3.12 bugs and security fixes 3.13 bugs and security fixes labels Dec 21, 2023
@gpshead
Copy link
Member

gpshead commented Dec 21, 2023

The check-cleanup-src target had a typo in its checks. It was failing to detect Python/frozen_modules/*.h files in the source directory tree. PR added.

gpshead added a commit that referenced this issue Dec 21, 2023
…13344)

A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Dec 21, 2023
…s. (pythonGH-113344)

A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
(cherry picked from commit 103c4ea)

Co-authored-by: Gregory P. Smith <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Dec 21, 2023
…s. (pythonGH-113344)

A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
(cherry picked from commit 103c4ea)

Co-authored-by: Gregory P. Smith <[email protected]>
@gpshead
Copy link
Member

gpshead commented Dec 21, 2023

backports set to auto-merge. I didn't try to figure out why I had clients in this state given I don't do in-tree builds as a habit... but it doesn't matter. transient leftover bits over time in my clients that mostly hide by default due to .gitignore? now gone. and identified in the future. :)

@gpshead gpshead closed this as completed Dec 21, 2023
gpshead added a commit that referenced this issue Dec 21, 2023
…es. (GH-113344) (#113347)

gh-112305: Fix check-clean-src to detect frozen_modules .h files. (GH-113344)

A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
(cherry picked from commit 103c4ea)

Co-authored-by: Gregory P. Smith <[email protected]>
gpshead added a commit that referenced this issue Dec 21, 2023
…es. (GH-113344) (#113346)

gh-112305: Fix check-clean-src to detect frozen_modules .h files. (GH-113344)

A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
(cherry picked from commit 103c4ea)

Co-authored-by: Gregory P. Smith <[email protected]>
ryan-duve pushed a commit to ryan-duve/cpython that referenced this issue Dec 26, 2023
…s. (python#113344)

A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
kulikjak pushed a commit to kulikjak/cpython that referenced this issue Jan 22, 2024
…s. (python#113344)

A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
…s. (python#113344)

A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
…s. (python#113344)

A typo left this check broken so many of us who do out-of-tree builds
were seeing strange failures due to bad `Python/frozen_modules/*.h`
files being picked up from the source tree and used at build time from
different Python versions leading to errors like:

`Fatal Python error: _PyImport_InitCore: failed to initialize importlib`

Or similar once our build got to an "invoke the interpreter"
bootstrapping step due to incorrect bytecode being embedded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 bugs and security fixes 3.13 bugs and security fixes build The build process and cross-build type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

4 participants