bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

gpshead · 2020-12-14T08:20:44Z

lib2to3 is effectively EOL, but it was never taught to parse positional only args ala PEP-570 that shipped in 3.8. This fixes that oversight.

Even though destined to wind up on PyPI instead of in the stdlib, this should help give it a proper sendoff and unblock things still using it under 3.8.x and 3.9.x from falling over on modern syntax.

Notably the blib2to3 within Black already backported these https://github.com/python/cpython/blob/3.8/Grammar/Grammar updates.

https://bugs.python.org/issue36541

This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs.

Lib/lib2to3/tests/test_parser.py

pablogsal

LGTM

I left a minor comment

miss-islington · 2020-12-14T17:10:14Z

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8, 3.9.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

…-23759) Add positional only args support to lib2to3 pgen2. This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs. Also add tests for complex *expr and **expr's. (cherry picked from commit 42c9f0f) Co-authored-by: Gregory P. Smith <[email protected]>

bedevere-bot · 2020-12-14T17:10:28Z

GH-23768 is a backport of this pull request to the 3.9 branch.

bedevere-bot · 2020-12-14T17:10:39Z

GH-23769 is a backport of this pull request to the 3.8 branch.

…-23759) Add positional only args support to lib2to3 pgen2. This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs. Also add tests for complex *expr and **expr's. (cherry picked from commit 42c9f0f) Co-authored-by: Gregory P. Smith <[email protected]>

Add positional only args support to lib2to3 pgen2. This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs. Also add tests for complex *expr and **expr's. (cherry picked from commit 42c9f0f) Co-authored-by: Gregory P. Smith <[email protected]>

bedevere-bot · 2020-12-14T20:18:17Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora LTO + PGO 3.8 has failed when building commit 06bfd03.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/277/builds/209) and take a look at the build logs.
Check if the failure is related to this commit (06bfd03) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/277/builds/209

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 1        
remote: Enumerating objects: 11, done.        
remote: Counting objects:   9% (1/11)        
remote: Counting objects:  18% (2/11)        
remote: Counting objects:  27% (3/11)        
remote: Counting objects:  36% (4/11)        
remote: Counting objects:  45% (5/11)        
remote: Counting objects:  54% (6/11)        
remote: Counting objects:  63% (7/11)        
remote: Counting objects:  72% (8/11)        
remote: Counting objects:  81% (9/11)        
remote: Counting objects:  90% (10/11)        
remote: Counting objects: 100% (11/11)        
remote: Counting objects: 100% (11/11), done.        
remote: Total 12 (delta 10), reused 10 (delta 10), pack-reused 1        
From https://github.com/python/cpython
 * branch                  3.8        -> FETCH_HEAD
Reset branch '3.8'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [Makefile:1778: clean] Error 1 (ignored)
Objects/longobject.c: In function ‘_PyLong_Frexp’:
Objects/longobject.c:2989:33: warning: ‘x_digits[0]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 2989 |                     x_digits[0] |= 1;
      |                                 ^~
Objects/longobject.c: In function ‘PyLong_AsDouble’:
Objects/longobject.c:2989:33: warning: ‘x_digits[0]’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 2989 |                     x_digits[0] |= 1;
      |                                 ^~
In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5692:25,
    inlined from ‘assemble’ at Python/compile.c:6031:18:
Python/compile.c:5646:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5646 |         *lnotab++ = k;
      |         ~~~~~~~~~~^~~
In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5692:25,
    inlined from ‘assemble’ at Python/compile.c:6031:18:
Python/compile.c:5646:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5646 |         *lnotab++ = k;
      |                   ^
In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5692:25,
    inlined from ‘assemble’ at Python/compile.c:6031:18:
Python/compile.c:5646:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5646 |         *lnotab++ = k;
      |                   ^
In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5692:25,
    inlined from ‘assemble’ at Python/compile.c:6031:18:
Python/compile.c:5646:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5646 |         *lnotab++ = k;
      |                   ^
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_float’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:5395:9: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 5395 |     d = PyOS_string_to_double(s, &endptr, PyExc_OverflowError);
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_string’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:5444:30: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 5444 |     if (len >= 2 && s[0] == s[len - 1] && (s[0] == '\'' || s[0] == '"')) {
      |                              ^
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_int’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:5181:17: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 5181 |         value = PyLong_FromString(s, NULL, 0);
      |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_get’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:6227:11: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 6227 |     key = PyLong_FromString(s, NULL, 10);
      |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c: In function ‘load_persid’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_pickle.c:6107:15: warning: ‘s’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 6107 |         pid = PyUnicode_DecodeASCII(s, len - 1, "strict");
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c: In function ‘PyInit__testcapi’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c:6328:5: warning: ‘tp_print’ is deprecated [-Wdeprecated-declarations]
 6328 |     MyList_Type.tp_print = 0;
      |     ^~~~~~~~~~~
In file included from ./Include/object.h:746,
                 from ./Include/pytime.h:6,
                 from ./Include/Python.h:85,
                 from /home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c:15:
./Include/cpython/object.h:260:30: note: declared here
  260 |     Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
      |                              ^~~~~~~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c: In function ‘test_buildvalue_issue38913’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/_testcapimodule.c:6463:15: warning: variable ‘res’ set but not used [-Wunused-but-set-variable]
 6463 |     PyObject *res;
      |               ^~~
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/socketmodule.c: In function ‘getsockaddrarg’:
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Modules/socketmodule.c:2316:9: warning: ‘strncpy’ specified bound 64 equals destination size [-Wstringop-truncation]
 2316 |         strncpy((char *)sa->salg_name, name, sizeof(sa->salg_name));
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Parser/listnode.c: In function ‘list1node’:
Parser/listnode.c:66:1: warning: ‘/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Parser/listnode.gcda’ profile count data file not found [-Wmissing-profile]
   66 | }
      | ^
Python/frozenmain.c: In function ‘Py_FrozenMain’:
Python/frozenmain.c:129:1: warning: ‘/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/frozenmain.gcda’ profile count data file not found [-Wmissing-profile]
  129 | }
      | ^
Python/pyfpe.c: In function ‘PyFPE_dummy’:
Python/pyfpe.c:15:1: warning: ‘/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pyfpe.gcda’ profile count data file not found [-Wmissing-profile]
   15 | }
      | ^
/usr/bin/ld: python.lto.o: in function `run_mod':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1178: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: python.lto.o: in function `symtable_add_def_helper':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `builtin_compile':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/bltinmodule.c:808: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: python.lto.o: in function `Py_CompileStringObject':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1254: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: python.lto.o: in function `symtable_lookup':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1009: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `symtable_record_directive':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1163: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `type_new':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Objects/typeobject.c:2504: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `symtable_visit_expr':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o:/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:578: python] Error 1
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: _testembed.lto.o: in function `run_mod':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1178: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: _testembed.lto.o: in function `symtable_add_def_helper':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o: in function `builtin_compile':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/bltinmodule.c:808: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: _testembed.lto.o: in function `Py_CompileStringObject':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1254: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: _testembed.lto.o: in function `symtable_lookup':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1009: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o: in function `symtable_record_directive':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1163: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o: in function `type_new':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Objects/typeobject.c:2504: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o: in function `symtable_visit_expr':
/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o:/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: undefined reference to `_Py_Mangle'
/usr/bin/ld: _testembed.lto.o:/home/dje/cpython-buildarea/3.8.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1023: more undefined references to `_Py_Mangle' follow
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:709: Programs/_testembed] Error 1
make: *** [Makefile:521: profile-opt] Error 2

…-23759) Add positional only args support to lib2to3 pgen2. This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs. Also add tests for complex *expr and **expr's.

gpshead added 2 commits December 14, 2020 07:58

Add positional only args support to lib2to3 pgen2.

16989da

This adds 3.8's PEP-570 support to lib2to3's pgen2. lib2to3, while being deprecated is still used by things to parse all versions of Python code today. We need it to support parsing modern 3.8 and 3.9 constructs.

Add tests for complex *expr and **expr's.

093b468

gpshead added type-bug An unexpected behavior, bug, or error tests Tests in the Lib/test dir needs backport to 3.8 needs backport to 3.9 only security fixes labels Dec 14, 2020

the-knights-who-say-ni added the CLA signed label Dec 14, 2020

bedevere-bot added the awaiting core review label Dec 14, 2020

NEWS via blurb

6e728b0

gpshead self-assigned this Dec 14, 2020

gpshead mentioned this pull request Dec 14, 2020

Python 3.8 support google/yapf#772

Closed

pablogsal reviewed Dec 14, 2020

View reviewed changes

Lib/lib2to3/tests/test_parser.py Show resolved Hide resolved

pablogsal approved these changes Dec 14, 2020

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Dec 14, 2020

Add another test case.

5d7321c

gpshead merged commit 42c9f0f into python:master Dec 14, 2020

bedevere-bot removed the awaiting merge label Dec 14, 2020

gpshead deleted the lib2to3-grammar-positional-only branch December 14, 2020 17:10

bedevere-bot removed needs backport to 3.9 only security fixes needs backport to 3.8 labels Dec 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

gpshead commented Dec 14, 2020 •

edited by bedevere-bot

Loading

pablogsal left a comment

miss-islington commented Dec 14, 2020

bedevere-bot commented Dec 14, 2020

bedevere-bot commented Dec 14, 2020

bedevere-bot commented Dec 14, 2020

bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

bpo-36541: Add lib2to3 grammar PEP-570 pos-only arg parsing #23759

Conversation

gpshead commented Dec 14, 2020 • edited by bedevere-bot Loading

pablogsal left a comment

Choose a reason for hiding this comment

miss-islington commented Dec 14, 2020

bedevere-bot commented Dec 14, 2020

bedevere-bot commented Dec 14, 2020

bedevere-bot commented Dec 14, 2020

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

gpshead commented Dec 14, 2020 •

edited by bedevere-bot

Loading