Skip to content

Micro-optimization: Make ArgKind a regular class instead of enum #19465

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

JukkaL
Copy link
Collaborator

@JukkaL JukkaL commented Jul 16, 2025

Mypyc doesn't generate very efficient code for enums, so switch to a regular class. We can later revert the change if/when we can improve enum support in mypyc.

Operations related to ArgKind were pretty prominent in the op trace log (#19457).

By itself this improves performance by ~1.7%, based on perf_compare.py, which is significant:

master                    4.168s (0.0%) | stdev 0.037s
HEAD                      4.098s (-1.7%) | stdev 0.028s

This is a part of a set of micro-optimizations that improve self check performance by ~5.5%.

Mypyc doesn't generate very efficient code for enums, so switch to a
regular class. We can later revert the change if/when we can improve
enum support in mypyc.

Operations related to ArgKind were pretty prominent in the op trace
log (#19457).

By itself this improves performance by ~1.7%, based on
`perf_compare.py`, which is significant:
```
master                    4.168s (0.0%) | stdev 0.037s
HEAD                      4.098s (-1.7%) | stdev 0.028s
```

This is a part of a set of micro-optimizations that improve self check
performance by ~5.5%.
@JukkaL JukkaL requested a review from ilevkivskyi July 16, 2025 13:50
Copy link
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

django-stubs (https://github.com/typeddjango/django-stubs)
+ django-stubs/contrib/sites/models.pyi:16: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
+ https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
+ Please report a bug at https://github.com/python/mypy/issues
+ version: 1.18.0+dev.3194db2e144e768800ecb9650111103a3f6e9bdf
+ django-stubs/contrib/sites/models.pyi:16: : note: use --pdb to drop into pdb
+ Traceback (most recent call last):
+   File "", line 8, in <module>
+     sys.exit(console_entry())
+   File "/__main__.py", line 15, in console_entry
+     main()
+   File "/main.py", line 127, in main
+     res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
+   File "/main.py", line 211, in run_build
+     res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
+   File "/build.py", line 191, in build
+     result = _build(
+   File "/build.py", line 267, in _build
+     graph = dispatch(sources, manager, stdout)
+   File "/build.py", line 2939, in dispatch
+     process_graph(graph, manager)
+   File "/build.py", line 3337, in process_graph
+     process_stale_scc(graph, scc, manager)
+   File "/build.py", line 3438, in process_stale_scc
+     graph[id].type_check_first_pass()
+   File "/build.py", line 2311, in type_check_first_pass
+     self.type_checker().check_first_pass()
+   File "/checker.py", line 483, in check_first_pass
+     self.accept(d)
+   File "/checker.py", line 590, in accept
+     stmt.accept(self)
+   File "/nodes.py", line 1259, in accept
+     return visitor.visit_class_def(self)
+   File "/checker.py", line 2560, in visit_class_def
+     self.accept(defn.defs)
+   File "/checker.py", line 590, in accept
+     stmt.accept(self)
+   File "/nodes.py", line 1340, in accept
+     return visitor.visit_block(self)
+   File "/checker.py", line 3021, in visit_block
+     self.accept(s)
+   File "/checker.py", line 590, in accept
+     stmt.accept(self)
+   File "/nodes.py", line 1427, in accept
+     return visitor.visit_assignment_stmt(self)
+   File "/checker.py", line 3070, in visit_assignment_stmt
+     self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
+   File "/checker.py", line 3282, in check_assignment
+     rvalue_type = self.expr_checker.accept(rvalue, type_context=type_context)
+   File "/checkexpr.py", line 5982, in accept
+     typ = node.accept(self)
+   File "/nodes.py", line 2080, in accept
+     return visitor.visit_call_expr(self)
+            ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
+   File "/checkexpr.py", line 520, in visit_call_expr
+     return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   File "/checkexpr.py", line 654, in visit_call_expr_inner
+     ret_type = self.check_call_expr_with_callee_type(
+         callee_type, e, fullname, object_type, member
+     )
+   File "/checkexpr.py", line 1504, in check_call_expr_with_callee_type
+     ret_type, callee_type = self.check_call(
+                             ~~~~~~~~~~~~~~~^
+         callee_type,
+         ^^^^^^^^^^^^
+     ...<6 lines>...
+         object_type=object_type,
+         ^^^^^^^^^^^^^^^^^^^^^^^^
+     )
+     ^
+   File "/checkexpr.py", line 1606, in check_call
+     return self.check_callable_call(
+            ~~~~~~~~~~~~~~~~~~~~~~~~^
+         callee,
+         ^^^^^^^
+     ...<6 lines>...
+         object_type,
+         ^^^^^^^^^^^^
+     )
+     ^
+   File "/checkexpr.py", line 1851, in check_callable_call
+     new_ret_type = self.apply_function_plugin(
+         callee,
+     ...<7 lines>...
+         context,
+     )
+   File "/checkexpr.py", line 1278, in apply_function_plugin
+     return callback(
+         FunctionContext(
+     ...<8 lines>...
+         )
+     )
+   File "_django_plugin/transformers/fields.py", line 255, in transform_into_proper_return_type
+     return set_descriptor_types_for_field_callback(ctx, django_context)
+   File "_django_plugin/transformers/fields.py", line 136, in set_descriptor_types_for_field_callback
+     return set_descriptor_types_for_field(ctx)
+   File "_django_plugin/transformers/fields.py", line 145, in set_descriptor_types_for_field
+     null_expr = helpers.get_call_argument_by_name(ctx, "null")
+   File "_django_plugin/lib/helpers.py", line 206, in get_call_argument_by_name
+     if kind == ArgKind.ARG_NAMED and argname == name:
+                ^^^^^^^^^^^^^^^^^
+ AttributeError: type object 'ArgKind' has no attribute 'ARG_NAMED'

@sterliakov
Copy link
Collaborator

How difficult would be the relevant changes to enum handling in mypyc? I can try to work on that. This PR incurs a huge maintainability and readability penalty IMO, but probably 1.7% is worth that:(

@JukkaL
Copy link
Collaborator Author

JukkaL commented Jul 16, 2025

Yeah this is pretty unfortunate. Since this change seems to break some plugins, it may be better to improve the enum support in mypyc instead and forget about this. I can create a mypyc issue detailing the requirements needed to make this better.

@JukkaL JukkaL closed this Jul 16, 2025
@ilevkivskyi
Copy link
Member

@sterliakov I think we can optimize enums in general in mypyc (e.g. use some static pointers for enum values, use pointer identity comparison for ==, and inline enum references when compiling, i.e. SomeEnum.some_val will become _CPy_SomeEnum_some_val). Most likely the perf benefit is from not having to go through attribute access logic, but TBH I didn't really think much about this.

That said I agree having optimized enums in mypyc is IMO worth it.

@JukkaL
Copy link
Collaborator Author

JukkaL commented Jul 18, 2025

@sterliakov @ilevkivskyi I created a mypyc issue about this: mypyc/mypyc#1121

Calling methods of enum classes is the biggest bottleneck that I can see. Accessing enum values is already optimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants