bpo-43822: Improve syntax errors for missing commas

[pablogsal]

https://bugs.python.org/issue43822

[pablogsal]

This PR also introduces the possibility to match against any soft keyword (to exclude the matches without having to list manually all soft keywords).

[gvanrossum]

Interesting. I guess this means that [a, b c] elicits this error but [a, match x] or [a, case x] doesn't. I guess that's fine though it's a bit of an overreach, since soft keywords are context-dependent.

[pablogsal]

Interesting. I guess this means that [a, b c] elicits this error but [a, match x] or [a, case x] doesn't. I guess that's fine though it's a bit of an overreach, since soft keywords are context-dependent.

Indeed, given that this is a hint, I find the compromise acceptable given that provides a good balance between maintainability (not a crazy rules to pick the correct context) and providing some value. If someone wants to refine this later, we can do it :)

[gvanrossum]

Agreed, it was just a casual remark. :-)

[pablogsal]

@lysnikolaou could you check this one if you have some time?

[lysnikolaou]

LG! 🚀

[lysnikolaou]

Why can't we use the KNOWN_LOCATION variant here?

Why can't we use the normal RAISE_SYNTAX_ERROR_KNOWN_LOCATION here?

[pablogsal]

The reason is because I want to point to the end of the expression, not to the begging of it (because that's where the comma should be).

(I deleted my previous comment because I forgot that this was the real reason)

[lysnikolaou]

Correct. It's good to go then.

(Now you've gotten me really interested though.)

[pablogsal]

@lysnikolaou 🤫🤫🤫🤫🤫

>>> f(a, b, x for x in range(10))
  File "<stdin>", line 1
    f(a, b, x for x in range(10))
            ^^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized

>>> blech bluch
  File "<stdin>", line 1
    blech bluch
          ^^^^^
SyntaxError: invalid syntax

>>> {1:2, 3: *3434, 3:4}
  File "<stdin>", line 1
    {1:2, 3: *3434, 3:4}
             ^^^^^
SyntaxError: cannot use a starred expression in a dictionary value

[pablogsal]

@lysnikolaou Actually, I may need help with this if we want to get it to beta freeze. Do you want to give me a hand?

[lysnikolaou]

Oh wow! Yup, count me in! How should we go about it?

[pablogsal]

Checkout my prototype: https://github.com/pablogsal/cpython/tree/tokenizer_better_errors

This basically adds a "end column offset" to syntax errors and threads it down everywhere.

The moderate bigger task is to make the parser/grammar to populate that accordingly. Now I am doing this hack:

https://github.com/pablogsal/cpython/blob/tokenizer_better_errors/Parser/pegen.c#L426

as a way to populate that.

Basically what's left is to productionalise this. There are several things to do:

• The hack doesn't work always (check one of the new dict errors like {1:2, 3, 3:3}.
• Devise a better way to make the parser inject this automatically and maybe a new macro so you can mark the beginning and the end passing two tokens instead of 1.
• Tests