Multiline expression brackets with format specifiers don't work in f-strings

[tusharsadhwani]

Bug report

Bug description:

In accordance with PEP 701, the following code works:

>>> x = 1
>>> f"___{
...     x
... }___"
'___1___'

>>> f"___{(
...     x
... )}___"
'___1___'

But the following fails:

f"__{
    x:d
}__"

This gives:

  File "<stdin>", line 1
    x:d
SyntaxError: unterminated f-string literal (detected at line 2)

Is this intended behaviour? This is not clarified in the PEP.

---

Similarly,

f"""__{
    x:d
}__"""

Gives:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier 'd
' for object of type 'int'

CPython versions tested on:

3.12

Operating systems tested on:

macOS

Linked PRs

• gh-110259: Fix f-strings with multiline expressions and format specs #110271
• [3.12] gh-110259: Fix f-strings with multiline expressions and format specs (GH-110271) #110396

[pablogsal]

Thanks for the report! I have opened #110271 to fix it

[ericvsmith]

I'd be careful on that last one. If x is a datetime.datetime, it's a valid format spec.

>>> x=datetime.datetime.now()
>>> f"""__{
...    x:d
... }__"""
'__d\n__'

[ericvsmith]

In case that's not clear: format specs for datetime.datetime can contain any characters, and are passed on to strftime. Perhaps more usefully:

>>> f"""__{
...     x:%Y
... %m
... %d
... }__"""
'__2023\n10\n03\n__'

[pablogsal]

Hummmm, then what you are saying is that we should not include the newline in the single quoted case but we should do it in the triple quoted case?

[pablogsal]

Ah, indeed, this is the case because the triple quote case works like that in 3.11:

Python 3.11.1 (main, Jan  3 2023, 20:00:19) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> f"""__{
...    datetime.datetime.now():%Y
...    %m
...    %d
... }__"""
'__2023\n   10\n   04\n__'

[lysnikolaou]

Hummmm, then what you are saying is that we should not include the newline in the single quoted case but we should do it in the triple quoted case?

I don't understand why this is the case. Shouldn't these both lead to the same AST?

Multi-line f-string:

f"""__{
    x:%Y
%m
%d
}__"""

Single-quoted f-string:

f"__{
    x:%Y
%m
%d
}__"

[ericvsmith]

I don't think it's necessarily an issue. I was just pointing out that the error shown in the original report

f"""__{
    x:d
}__"""

(a ValueError from the bad format spec "d\n" for int) is different from the error in

f"__{
    x:d
}__"

(a SyntaxError). In any event, the format spec should be allowed to have newlines, including trailing newlines.

[tusharsadhwani]

@ericvsmith the triple quoted issue is indeed different from the single quoted case.

But, is that a bug?

f"""{
  5:.2f
}"""

doesn't work on 3.11 and below as well, but I feel like it should.

[ericvsmith]

@tusharsadhwani Maybe I could be talked into changing it, but the change would have to be in float.__format__ to ignore trailing newlines (or maybe all trailing whitespace?). So, it's unrelated to the original SyntaxError reported here. And if we do change float.__format__, the same change should be made for int, str, complex, and Decimal. I think (but won't swear to it} that those are the only built-in types that handle the format spec "mini-language" defined in https://docs.python.org/3/library/string.html#format-specification-mini-language. For types implemented in C that use the mini-language, there is a common function for parsing format specs, so they'd all pick it up "for free". I don't recall if Decimal uses it or if it has its own function for that.

This change cannot be in the parser, because whether or not whitespace in the format spec is an error or is acceptable is up to the type being formatted. As things stand now, datetime.datetime.__format__ allows the whitespace, and float.__format__ and others do not.

[tusharsadhwani]

Agreed.

The change for single quotes should still go through IMO, because the AST should be identical for the formatted brackets even if the fstring gets split on multiple lines. For triple quotes the ASTs must be different because of backwards compatibility reasons.

[pablogsal]

There is still ambiguity on how to treat the single quote case. The most natural way for the tokenizer is to stop the format string on the first \n because the string chunks of single quote f-strings cannot spawn more than one line. So this means that this is valid:

>>> f"__{
... 1
... +
... 1:d
...
...
... }__"

but this is not:

>>> f"__{
... 1
... +
... 1:a
... b
...
... }__"

The other option is allowing multi-line format specs but that may be a bit more annoying to implement. @ericvsmith @lysnikolaou what do you think?

[pablogsal]

PR #110271 implements single-line format specifiers for single quoted f-strings, but we can change it if we think multi-line specs should be allowed.

[tusharsadhwani]

It seems like an arbitrary limitation on code formatters: splitting any f-string on the expression brackets will be valid, unless it's the specific case of a single quoted fstring with a format specifier.

[pablogsal]

Not really, you can still be able to split the f-string in the brackets, but not the format specifier. This is fine because if you split the format specifier you are changing the f-string anyway.

To clarify, you can change:

f"__{1+1:xxx}__"

to

f"__{
1
+
1:xxx


}__"

if you want, but you cannot split the xxx

[tusharsadhwani]

Makes sense, the linked PR seems good to me

[pablogsal]

So basically, in other words, if single quoted f-strings format specifiers cannot spawn multiple lines themselves.

[ericvsmith]

@pablogsal So I think you're saying that for single quoted f-strings, there can't be a newline in the format spec. Although it looks like that's a change, I think it's okay.

ETA: Our messages crossed. I think we're in agreement.

[lysnikolaou]

I think that's okay as well. Tokenizing multi-line format specifiers in single-quoted f-strings would be a bit cumbersome, so let's go with that.

[pablogsal]

ETA: Our messages crossed. I think we're in agreement.

Sorry, I am also on a plane, so the timing of my messages may be a bit inconsistent :P

[pablogsal]

Update: i tried to check how annoying would be to support the other option (support multi-line format specifiers) and turns out that is quite easy to support thanks to how we have structured this (yay!). So only if we want we can support that.

It would look like this, for clarification:

code = """\
f"__{
1
+
1:a
b
c

}__"
"""

print(ast.dump(ast.parse(code), indent=4))

which prints:

Module(
    body=[
        Expr(
            value=JoinedStr(
                values=[
                    Constant(value='__'),
                    FormattedValue(
                        value=BinOp(
                            left=Constant(value=1),
                            op=Add(),
                            right=Constant(value=1)),
                        conversion=-1,
                        format_spec=JoinedStr(
                            values=[
                                Constant(value='a\nb\nc\n\n')])),
                    Constant(value='__')]))],
    type_ignores=[])

Sorry for the back and forth @lysnikolaou @ericvsmith. I think we are free to choose whatever we prefer.

[pablogsal]

Notice that this has an important difference that I think it makes is less appealing: extra trailing newlines are included in the format specifier, which I think it will make formatters unhappy (@tusharsadhwani, please confirm)

[tusharsadhwani]

Can confirm, second option will restrict code formatting, pretty much making fstrings with format specs unformattable.

But, fstrings don't get formatted by black right now as-is and there's no confirmation on my end that that's desired.

[ericvsmith]

@tusharsadhwani black can't reformat the format specs without breaking datetime.datetime (for example).

@pablogsal I don't think it matters all that much. I can't imagine that there are many single quote multi-line format specs, although they no doubt exist.

[pablogsal]

Ok, I ended implementing the single-line restruction for format specs in single-quoted f-strings. Thanks everyone for your help, you rock 🚀

[tusharsadhwani]

@ericvsmith wasn't single quote multiline format spec invalid syntax until now?

[ericvsmith]

@tusharsadhwani : I thought somewhere on this thread someone said it worked in 3.11, but I guess I misread that.