`compile` fails on 3.14 on a valid expression when `-OO` is set

[tybug]

Bug report

Bug description:

Given the following code snippet:

import ast

source = b'class A:\n    """\n    """\n'
compile(ast.parse(source), "a", "exec")

python3.13 -OO test.py passes, but python3.14 -OO test.py fails, with:

Traceback (most recent call last):
  File "/Users/tybug/Desktop/sandbox2.py", line 7, in <module>
    compile(ast.parse(source), "a", "exec")
ValueError: empty body on ClassDef

Python: 3.14.0rc1

CPython versions tested on:

3.14

Operating systems tested on:

macOS

Linked PRs

• gh-137308: Replace a single docstring with pass #137318
• [3.14] gh-137308: Replace a single docstring with pass in -OO mode (GH-137318) #137322

[StanFromIreland]

CC @sobolevn (due to -OO)

The docstring is optimised away here, I’m not sure why it was allowed in 3.13.

[picnixz]

I think we somehow changed the parser.

import ast
stmt = "class A:\n\t\"\"\"\n\t\"\"\"\n"
print(ast.dump(ast.parse(stmt)))

With Python 3.12:

Module(body=[ClassDef(name='A', bases=[], keywords=[], body=[Expr(value=Constant(value='\n\t'))], decorator_list=[], type_params=[])], type_ignores=[])

With main:

Module(body=[ClassDef(name='A')])

EDIT:

Apparently, it's more than that. The docstring is not properly eliminated prior to Python 3.14:

$ python3.13 -OO -c 'import ast; stmt = "class A:\n\t\"\"\"abc\"\"\"\n"; print(ast.dump(ast.parse(stmt, optimize=2)))'
Module(body=[ClassDef(name='A', body=[Expr(value=Constant(value='abc'))])])

[brianschubert]

Bisects to ffc2f1d (#130087), cc @iritkatriel

[picnixz]

I think it's because when we parse the AST we somehow also optimize it, namely we inherit from "-OO":

$ # at 468a7aaeb4dfcd21697dfa46362b9f415b9daf1b (before ffc2f1d)
$ python -OO -c 'import ast; stmt = "class A:\n\t\"\"\"abc\"\"\"\n"; print(ast.dump(ast.parse(stmt)))'
Module(body=[ClassDef(name='A', body=[Expr(value=Constant(value='abc'))])])
$ python -OO -c 'import ast; stmt = "class A:\n\t\"\"\"abc\"\"\"\n"; print(ast.dump(ast.parse(stmt, optimize=0)))'
Module(body=[ClassDef(name='A', body=[Expr(value=Constant(value='abc'))])])
$ python -OO -c 'import ast; stmt = "class A:\n\t\"\"\"abc\"\"\"\n"; print(ast.dump(ast.parse(stmt, optimize=1)))'
Module(body=[ClassDef(name='A', body=[Expr(value=Constant(value='abc'))])])
$ python -OO -c 'import ast; stmt = "class A:\n\t\"\"\"abc\"\"\"\n"; print(ast.dump(ast.parse(stmt, optimize=2)))'
Module(body=[ClassDef(name='A')])

$ # at 9a21df7c0a494e2819775eabd522ebec994d96c0 (roughly now)
$ python -OO -c 'import ast; stmt = "class A:\n\t\"\"\"abc\"\"\"\n"; print(ast.dump(ast.parse(stmt)))'
Module(body=[ClassDef(name='A')])
$ python -OO -c 'import ast; stmt = "class A:\n\t\"\"\"abc\"\"\"\n"; print(ast.dump(ast.parse(stmt, optimize=0)))'
Module(body=[ClassDef(name='A', body=[Expr(value=Constant(value='abc'))])])
$ python -OO -c 'import ast; stmt = "class A:\n\t\"\"\"abc\"\"\"\n"; print(ast.dump(ast.parse(stmt, optimize=1)))'
Module(body=[ClassDef(name='A', body=[Expr(value=Constant(value='abc'))])])
$ python -OO -c 'import ast; stmt = "class A:\n\t\"\"\"abc\"\"\"\n"; print(ast.dump(ast.parse(stmt, optimize=2)))'
Module(body=[ClassDef(name='A')])

So the bisected commit actually had a side-effect where ast.parse() seems to inherit the value of optimize from the interpreter [it should per the docs but it didn't seem to be the case previously] (it's only inherited by default, not when we explicitly indicate optimize=...)

EDIT:

Let's quote the ast.parse docs again:

Parse the source into an AST node. Equivalent to compile(source, filename, mode,
flags=FLAGS_VALUE, optimize=optimize), where FLAGS_VALUE is ast.PyCF_ONLY_AST if
optimize <= 0 and ast.PyCF_OPTIMIZED_AST otherwise.

I think the issue is how we handle the optimize default value and when we actually handle it (and how we then determine the flags to pass). In Python 3.13 it seems that the flags and optimize values passed to compile after calling ast.parse were not the correct ones (I haven't checked) and incidently, they are now correct. Note that AST optimization (defined via ast.PyCF_*( has nothing to do with optimize=... which is about parser optimization as one could still have docstrings even if we had ast.PyCF_OPTIMIZED_AST. Docstrings disappear once we have optimize=2 or -OO.

[picnixz]

More generally, eliminating the docstring makes the class def (or the function def) empty. A similar error occurs with a function. Maybe docstring elimination should always add a "pass" statement instead? (namely we replace docstrings by pass)