-
-
Notifications
You must be signed in to change notification settings - Fork 3k
Micro-optimization: Make ArgKind a regular class instead of enum #19465
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Mypyc doesn't generate very efficient code for enums, so switch to a regular class. We can later revert the change if/when we can improve enum support in mypyc. Operations related to ArgKind were pretty prominent in the op trace log (#19457). By itself this improves performance by ~1.7%, based on `perf_compare.py`, which is significant: ``` master 4.168s (0.0%) | stdev 0.037s HEAD 4.098s (-1.7%) | stdev 0.028s ``` This is a part of a set of micro-optimizations that improve self check performance by ~5.5%.
Diff from mypy_primer, showing the effect of this PR on open source code: django-stubs (https://github.com/typeddjango/django-stubs)
+ django-stubs/contrib/sites/models.pyi:16: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
+ https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
+ Please report a bug at https://github.com/python/mypy/issues
+ version: 1.18.0+dev.3194db2e144e768800ecb9650111103a3f6e9bdf
+ django-stubs/contrib/sites/models.pyi:16: : note: use --pdb to drop into pdb
+ Traceback (most recent call last):
+ File "", line 8, in <module>
+ sys.exit(console_entry())
+ File "/__main__.py", line 15, in console_entry
+ main()
+ File "/main.py", line 127, in main
+ res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
+ File "/main.py", line 211, in run_build
+ res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
+ File "/build.py", line 191, in build
+ result = _build(
+ File "/build.py", line 267, in _build
+ graph = dispatch(sources, manager, stdout)
+ File "/build.py", line 2939, in dispatch
+ process_graph(graph, manager)
+ File "/build.py", line 3337, in process_graph
+ process_stale_scc(graph, scc, manager)
+ File "/build.py", line 3438, in process_stale_scc
+ graph[id].type_check_first_pass()
+ File "/build.py", line 2311, in type_check_first_pass
+ self.type_checker().check_first_pass()
+ File "/checker.py", line 483, in check_first_pass
+ self.accept(d)
+ File "/checker.py", line 590, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1259, in accept
+ return visitor.visit_class_def(self)
+ File "/checker.py", line 2560, in visit_class_def
+ self.accept(defn.defs)
+ File "/checker.py", line 590, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1340, in accept
+ return visitor.visit_block(self)
+ File "/checker.py", line 3021, in visit_block
+ self.accept(s)
+ File "/checker.py", line 590, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1427, in accept
+ return visitor.visit_assignment_stmt(self)
+ File "/checker.py", line 3070, in visit_assignment_stmt
+ self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
+ File "/checker.py", line 3282, in check_assignment
+ rvalue_type = self.expr_checker.accept(rvalue, type_context=type_context)
+ File "/checkexpr.py", line 5982, in accept
+ typ = node.accept(self)
+ File "/nodes.py", line 2080, in accept
+ return visitor.visit_call_expr(self)
+ ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
+ File "/checkexpr.py", line 520, in visit_call_expr
+ return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ File "/checkexpr.py", line 654, in visit_call_expr_inner
+ ret_type = self.check_call_expr_with_callee_type(
+ callee_type, e, fullname, object_type, member
+ )
+ File "/checkexpr.py", line 1504, in check_call_expr_with_callee_type
+ ret_type, callee_type = self.check_call(
+ ~~~~~~~~~~~~~~~^
+ callee_type,
+ ^^^^^^^^^^^^
+ ...<6 lines>...
+ object_type=object_type,
+ ^^^^^^^^^^^^^^^^^^^^^^^^
+ )
+ ^
+ File "/checkexpr.py", line 1606, in check_call
+ return self.check_callable_call(
+ ~~~~~~~~~~~~~~~~~~~~~~~~^
+ callee,
+ ^^^^^^^
+ ...<6 lines>...
+ object_type,
+ ^^^^^^^^^^^^
+ )
+ ^
+ File "/checkexpr.py", line 1851, in check_callable_call
+ new_ret_type = self.apply_function_plugin(
+ callee,
+ ...<7 lines>...
+ context,
+ )
+ File "/checkexpr.py", line 1278, in apply_function_plugin
+ return callback(
+ FunctionContext(
+ ...<8 lines>...
+ )
+ )
+ File "_django_plugin/transformers/fields.py", line 255, in transform_into_proper_return_type
+ return set_descriptor_types_for_field_callback(ctx, django_context)
+ File "_django_plugin/transformers/fields.py", line 136, in set_descriptor_types_for_field_callback
+ return set_descriptor_types_for_field(ctx)
+ File "_django_plugin/transformers/fields.py", line 145, in set_descriptor_types_for_field
+ null_expr = helpers.get_call_argument_by_name(ctx, "null")
+ File "_django_plugin/lib/helpers.py", line 206, in get_call_argument_by_name
+ if kind == ArgKind.ARG_NAMED and argname == name:
+ ^^^^^^^^^^^^^^^^^
+ AttributeError: type object 'ArgKind' has no attribute 'ARG_NAMED'
|
How difficult would be the relevant changes to enum handling in mypyc? I can try to work on that. This PR incurs a huge maintainability and readability penalty IMO, but probably 1.7% is worth that:( |
Yeah this is pretty unfortunate. Since this change seems to break some plugins, it may be better to improve the enum support in mypyc instead and forget about this. I can create a mypyc issue detailing the requirements needed to make this better. |
@sterliakov I think we can optimize enums in general in mypyc (e.g. use some static pointers for enum values, use pointer identity comparison for That said I agree having optimized enums in mypyc is IMO worth it. |
@sterliakov @ilevkivskyi I created a mypyc issue about this: mypyc/mypyc#1121 Calling methods of enum classes is the biggest bottleneck that I can see. Accessing enum values is already optimized. |
Mypyc doesn't generate very efficient code for enums, so switch to a regular class. We can later revert the change if/when we can improve enum support in mypyc.
Operations related to ArgKind were pretty prominent in the op trace log (#19457).
By itself this improves performance by ~1.7%, based on
perf_compare.py
, which is significant:This is a part of a set of micro-optimizations that improve self check performance by ~5.5%.