bpo-39769: Fix compileall ddir for subpkgs. #18676

gpshead · 2020-02-27T08:47:17Z

Fixes compileall.compile_dir()'s ddir parameter and compileall command
line flag -d to no longer write the wrong pathname to the generated
pyc file for submodules beneath the root of the directory tree being
compiled. This fixes a regression introduced with Python 3.5.

https://bugs.python.org/issue39769

Fixes compileall.compile_dir's ddir parameter and compileall command line flag `-d` to no longer write the wrong pathname to the generated pyc file for submodules beneath the root of the directory tree being compiled. This fixes a regression introduced with Python 3.5.

gpshead · 2020-02-27T08:49:55Z

while i marked this as needing a backport... it'll never be automatic. This fix was easy to make as I could use the new features introduced by #16012 to implement it. I'm not sure it is worth it, nobody noticed for four releases. Easiest to just let 3.9 get this.

codecov · 2020-02-27T09:28:29Z

Codecov Report

Merging #18676 into master will increase coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #18676     +/-   ##
=========================================
  Coverage   82.12%   82.13%             
=========================================
  Files        1956     1955      -1     
  Lines      589830   584531   -5299     
  Branches    44477    44481      +4     
=========================================
- Hits       484427   480096   -4331     
+ Misses      95752    94790    -962     
+ Partials     9651     9645      -6

Impacted Files	Coverage Δ
Lib/distutils/tests/test_bdist_rpm.py	`30.00% <0.00%> (-65.00%)`	⬇️
Lib/distutils/command/bdist_rpm.py	`7.63% <0.00%> (-56.88%)`	⬇️
Modules/_decimal/libmpdec/umodarith.h	`80.76% <0.00%> (-19.24%)`	⬇️
Lib/test/test_urllib2net.py	`76.92% <0.00%> (-13.85%)`	⬇️
Lib/test/test_smtpnet.py	`78.57% <0.00%> (-7.15%)`	⬇️
Lib/ftplib.py	`63.85% <0.00%> (-6.06%)`	⬇️
Lib/test/test_ftplib.py	`87.11% <0.00%> (-4.72%)`	⬇️
Tools/scripts/db2pickle.py	`17.82% <0.00%> (-3.97%)`	⬇️
Tools/scripts/pickle2db.py	`16.98% <0.00%> (-3.78%)`	⬇️
Lib/test/test_socket.py	`71.94% <0.00%> (-3.77%)`	⬇️
... and 328 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 02a4d57...4e3543f. Read the comment docs.

Lib/compileall.py

Lib/test/test_compileall.py

bedevere-bot · 2020-02-27T21:44:45Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

gpshead · 2020-02-28T05:57:50Z

I have made the requested changes; please review again.

bedevere-bot · 2020-02-28T05:57:53Z

Thanks for making the requested changes!

@Yhg1s: please review the changes made to this pull request.

miss-islington · 2020-02-29T01:28:40Z

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington · 2020-02-29T01:28:43Z

Sorry @gpshead, I had trouble checking out the 3.8 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 02673352b5db6ca4d3dc804965facbedfe66425d 3.8

Fix compileall.compile_dir() ddir= behavior on sub-packages. Fixes compileall.compile_dir's ddir parameter and compileall command line flag `-d` to no longer write the wrong pathname to the generated pyc file for submodules beneath the root of the directory tree being compiled. This fixes a regression introduced with Python 3.5. Tests backported from GH 0267335, the implementation is different due to intervening code changes. But still quiet simple. The refactoring to add parallel execution kept the ddir -> dfile computations but discarded the results instead of sending them to compile_file(). This fixes that.

Fix compileall.compile_dir() ddir= behavior on sub-packages. Fixes compileall.compile_dir's ddir parameter and compileall command line flag `-d` to no longer write the wrong pathname to the generated pyc file for submodules beneath the root of the directory tree being compiled. This fixes a regression introduced with Python 3.5. Tests backported from GH 0267335, the implementation is different due to intervening code changes. But still quiet simple. Why was the bug ever introduced? The refactoring to add parallel execution kept the ddir -> dfile computations but discarded the results instead of sending them to compile_file(). This fixes that. Lack of tests meant this went unnoticed.

…ythonGH-18718) Fix compileall.compile_dir() ddir= behavior on sub-packages. Fixes compileall.compile_dir's ddir parameter and compileall command line flag `-d` to no longer write the wrong pathname to the generated pyc file for submodules beneath the root of the directory tree being compiled. This fixes a regression introduced with Python 3.5. Tests backported from GH 0267335, the implementation is different due to intervening code changes. But still quiet simple. Why was the bug ever introduced? The refactoring to add parallel execution kept the ddir -> dfile computations but discarded the results instead of sending them to compile_file(). This fixes that. Lack of tests meant this went unnoticed.. (cherry picked from commit ce720d3) Co-authored-by: Gregory P. Smith <[email protected]>

… (GH-18725) Fix compileall.compile_dir() ddir= behavior on sub-packages. Fixes compileall.compile_dir's ddir parameter and compileall command line flag `-d` to no longer write the wrong pathname to the generated pyc file for submodules beneath the root of the directory tree being compiled. This fixes a regression introduced with Python 3.5. Tests backported from GH 0267335, the implementation is different due to intervening code changes. But still quiet simple. Why was the bug ever introduced? The refactoring to add parallel execution kept the ddir -> dfile computations but discarded the results instead of sending them to compile_file(). This fixes that. Lack of tests meant this went unnoticed.. (cherry picked from commit ce720d3) Co-authored-by: Gregory P. Smith <[email protected]> [Google]

bedevere-bot · 2020-03-01T21:43:35Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora LTO + PGO 3.7 has failed when building commit 7c64726.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/541/builds/12) and take a look at the build logs.
Check if the failure is related to this commit (7c64726) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/541/builds/12

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 11, done.        
remote: Counting objects:   9% (1/11)        
remote: Counting objects:  18% (2/11)        
remote: Counting objects:  27% (3/11)        
remote: Counting objects:  36% (4/11)        
remote: Counting objects:  45% (5/11)        
remote: Counting objects:  54% (6/11)        
remote: Counting objects:  63% (7/11)        
remote: Counting objects:  72% (8/11)        
remote: Counting objects:  81% (9/11)        
remote: Counting objects:  90% (10/11)        
remote: Counting objects: 100% (11/11)        
remote: Counting objects: 100% (11/11), done.        
remote: Total 13 (delta 11), reused 11 (delta 11), pack-reused 2        
From https://github.com/python/cpython
 * branch                  3.7        -> FETCH_HEAD
Reset branch '3.7'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [Makefile:1722: clean] Error 1 (ignored)
In file included from /home/dje/cpython-buildarea/3.7.edelsohn-fedora-z.lto-pgo/build/Modules/expat/expat_config.h:8,
                 from /home/dje/cpython-buildarea/3.7.edelsohn-fedora-z.lto-pgo/build/Modules/expat/xmltok.c:49:
./pyconfig.h:1526: warning: "_POSIX_C_SOURCE" redefined
 1526 | #define _POSIX_C_SOURCE 200809L
      | 
In file included from /usr/include/bits/libc-header-start.h:33,
                 from /usr/include/string.h:26,
                 from /home/dje/cpython-buildarea/3.7.edelsohn-fedora-z.lto-pgo/build/Modules/expat/xmltok.c:34:
/usr/include/features.h:295: note: this is the location of the previous definition
  295 | # define _POSIX_C_SOURCE 199506L
      | 
Task was destroyed but it is pending!
task: <Task pending coro=<<async_generator_athrow without __name__>()>>
Task was destroyed but it is pending!
task: <Task pending coro=<<async_generator_athrow without __name__>()>>

https://bugs.python.org/issue39769 python/cpython#18676

the-knights-who-say-ni added the CLA signed label Feb 27, 2020

bedevere-bot added the awaiting core review label Feb 27, 2020

gpshead added needs backport to 3.8 type-bug An unexpected behavior, bug, or error and removed awaiting core review labels Feb 27, 2020

gpshead self-assigned this Feb 27, 2020

gpshead requested a review from Yhg1s February 27, 2020 08:51

brettcannon self-requested a review February 27, 2020 20:43

Yhg1s requested changes Feb 27, 2020

View reviewed changes

Lib/compileall.py Show resolved Hide resolved

Lib/compileall.py Outdated Show resolved Hide resolved

Lib/test/test_compileall.py Outdated Show resolved Hide resolved

Lib/test/test_compileall.py Outdated Show resolved Hide resolved

bedevere-bot added the awaiting changes label Feb 27, 2020

gpshead added 3 commits February 27, 2020 13:45

Add a function to read code from a pyc to test_importlib.util.

34e354b

Test both single and parallel workers.

6b5d576

Address code review comments.

dd244d3

bedevere-bot added awaiting change review and removed awaiting changes labels Feb 28, 2020

bedevere-bot requested a review from Yhg1s February 28, 2020 05:57

explicitly test ddir=''

4e3543f

Yhg1s approved these changes Feb 29, 2020

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels Feb 29, 2020

gpshead merged commit 0267335 into python:master Feb 29, 2020

bedevere-bot removed the awaiting merge label Feb 29, 2020

gpshead deleted the compileall_ddir branch February 29, 2020 01:28

gpshead mentioned this pull request Feb 29, 2020

[3.8] bpo-39769: Fix compileall ddir for subpkgs. (GH-18676) #18718

Merged

gpshead removed the needs backport to 3.8 label Feb 29, 2020

frenzymadness added a commit to fedora-python/compileall2 that referenced this pull request Mar 2, 2020

Backport fixes from bpo-39769

e8099f2

https://bugs.python.org/issue39769 python/cpython#18676

frenzymadness mentioned this pull request Mar 2, 2020

Backport fixes from bpo-39769 fedora-python/compileall2#23

Merged

hetmankp mentioned this pull request Oct 4, 2022

Regression in compileall ddir parameter when recursing #82896

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-39769: Fix compileall ddir for subpkgs. #18676

bpo-39769: Fix compileall ddir for subpkgs. #18676

gpshead commented Feb 27, 2020 •

edited by bedevere-bot

Loading

gpshead commented Feb 27, 2020 •

edited

Loading

codecov bot commented Feb 27, 2020 •

edited

Loading

bedevere-bot commented Feb 27, 2020

gpshead commented Feb 28, 2020

bedevere-bot commented Feb 28, 2020

miss-islington commented Feb 29, 2020

miss-islington commented Feb 29, 2020

bedevere-bot commented Mar 1, 2020

bpo-39769: Fix compileall ddir for subpkgs. #18676

bpo-39769: Fix compileall ddir for subpkgs. #18676

Conversation

gpshead commented Feb 27, 2020 • edited by bedevere-bot Loading

gpshead commented Feb 27, 2020 • edited Loading

codecov bot commented Feb 27, 2020 • edited Loading

Codecov Report

bedevere-bot commented Feb 27, 2020

gpshead commented Feb 28, 2020

bedevere-bot commented Feb 28, 2020

miss-islington commented Feb 29, 2020

miss-islington commented Feb 29, 2020

bedevere-bot commented Mar 1, 2020

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

gpshead commented Feb 27, 2020 •

edited by bedevere-bot

Loading

gpshead commented Feb 27, 2020 •

edited

Loading

codecov bot commented Feb 27, 2020 •

edited

Loading