Introduction to Apache Calcite
- 1. >< INTRODUCTIONTO INTRODUCTION TO APACHE CALCITE APACHE CALCITE JORDAN HALTERMAN 1
- 3. ><INTRODUCTION TO APACHE CALCITE 3 What is Apache Calcite? • A framework for building SQL databases • Developed over more than ten years • Written in Java • Previously known as Optiq • Previously known as Farrago • Became an Apache project in 2013 • Led by Julian Hyde at Hortonworks
- 4. ><INTRODUCTION TO APACHE CALCITE 4 Projects using Calcite • Apache Hive • Apache Drill • Apache Flink • Apache Phoenix • Apache Samza • Apache Storm • Apache everything…
- 5. ><INTRODUCTION TO APACHE CALCITE 5 What is Apache Calcite? • SQL parser • SQL validation • Query optimizer • SQL generator • Data federator
- 6. ><INTRODUCTION TO APACHE CALCITE Parse Queries are parsed using a JavaCC generated parser Validate Queries are validated against known database metadata Optimize Logical plans are optimized and converted into physical expressions Execute P h y s i c a l p l a n s a r e converted into application- specific executions 01 02 03 04 Stages of query execution 6
- 8. ><INTRODUCTION TO APACHE CALCITE 8 Components of Calcite • Catalog - Defines metadata and namespaces that can be accessed in SQL queries • SQL parser - Parses valid SQL queries into an abstract syntax tree (AST) • SQL validator - Validates abstract syntax trees against metadata provided by the catalog • Query optimizer - Converts AST into logical plans, optimizes logical plans, and converts logical expressions into physical plans • SQL generator - Converts physical plans to SQL
- 10. ><INTRODUCTION TO APACHE CALCITE 10 Calcite Catalog • Defines namespaces that can be accessed in Calcite queries • Schema • A collection of schemas and tables • Can be arbitrarily nested • Table • Represents a single data set • Fields defined by a RelDataType • RelDataType • Represents fields in a data set • Supports all SQL data types, including structs and
- 11. ><INTRODUCTION TO APACHE CALCITE 11 Schema • A collection of schemas and tables • Schemas can be arbitrarily nested
- 12. ><INTRODUCTION TO APACHE CALCITE 12 Schema • A collection of schemas and tables • Schemas can be arbitrarily nested
- 13. ><INTRODUCTION TO APACHE CALCITE 13 Table • Represents a single data set • Fields are defined by a RelDataType
- 14. ><INTRODUCTION TO APACHE CALCITE 14 Table • Represents a single data set • Fields are defined by a RelDataType
- 15. ><INTRODUCTION TO APACHE CALCITE 15 RelDataType • Represents the data type of an object • Supports all SQL data types, including structs and arrays • Similar to Spark’s DataType
- 16. ><INTRODUCTION TO APACHE CALCITE 16 RelDataType
- 17. ><INTRODUCTION TO APACHE CALCITE 17 RelDataType data type enum
- 18. ><INTRODUCTION TO APACHE CALCITE 18 Statistic • Provide table statistics used in optimization
- 19. ><INTRODUCTION TO APACHE CALCITE 19 Statistic • Provide table statistics used in optimization
- 20. ><INTRODUCTION TO APACHE CALCITE 20 Usage of the Calcite catalog
- 21. ><INTRODUCTION TO APACHE CALCITE 21 Usage of the Calcite catalog schema
- 22. ><INTRODUCTION TO APACHE CALCITE 22 Usage of the Calcite catalog schema table
- 23. ><INTRODUCTION TO APACHE CALCITE 23 Usage of the Calcite catalog schema table data type
- 24. ><INTRODUCTION TO APACHE CALCITE 24 Usage of the Calcite catalog schema table data typedata type field
- 25. SQL PARSER next 25
- 26. ><INTRODUCTION TO APACHE CALCITE 26 Calcite SQL parser • LL(k) parser written in JavaCC • Input queries are parsed into an abstract syntax tree (AST) • Tokens are represented in Calcite by SqlNode • SqlNode can also be converted back to a SQL string via the unparse method
- 27. ><INTRODUCTION TO APACHE CALCITE 27 JavaCC • Java Compiler Compiler • Created in 1996 at Sun Microsystems • Generates Java code from a domain- specific language • ANTLR is the modern alternative used in projects like Hive and Drill • JavaCC has sparse documentation
- 28. ><INTRODUCTION TO APACHE CALCITE 28 JavaCC
- 29. ><INTRODUCTION TO APACHE CALCITE 29 JavaCC
- 30. ><INTRODUCTION TO APACHE CALCITE 30 JavaCC tokens
- 31. ><INTRODUCTION TO APACHE CALCITE 31 JavaCC tokens or
- 32. ><INTRODUCTION TO APACHE CALCITE 32 JavaCC tokens or function call
- 33. ><INTRODUCTION TO APACHE CALCITE 33 JavaCC tokens Java code or function call
- 34. ><INTRODUCTION TO APACHE CALCITE 34 SqlNode • SqlNode represents an element in an abstract syntax tree
- 35. ><INTRODUCTION TO APACHE CALCITE 35 SqlNode • SqlNode represents an element in an abstract syntax tree select
- 36. ><INTRODUCTION TO APACHE CALCITE 36 SqlNode • SqlNode represents an element in an abstract syntax tree identifiersselect
- 37. ><INTRODUCTION TO APACHE CALCITE 37 SqlNode • SqlNode represents an element in an abstract syntax tree identifiersselect operator
- 38. ><INTRODUCTION TO APACHE CALCITE 38 SqlNode • SqlNode represents an element in an abstract syntax tree identifiersselect operator identifier
- 39. ><INTRODUCTION TO APACHE CALCITE 39 SqlNode • SqlNode represents an element in an abstract syntax tree identifiersselect operator identifier data type
- 40. ><INTRODUCTION TO APACHE CALCITE 40 SqlNode • SqlNode represents an element in an abstract syntax tree identifiersselect operator identifier data type identifier
- 41. ><INTRODUCTION TO APACHE CALCITE 41 SqlNode • SqlNode’s unparse method converts a SQL element back into a string
- 42. ><INTRODUCTION TO APACHE CALCITE 42 SqlNode • SqlNode’s unparse method converts a SQL element back into a string
- 43. ><INTRODUCTION TO APACHE CALCITE 43 SqlNode
- 44. ><INTRODUCTION TO APACHE CALCITE 44 SqlNode • SqlDialect indicates the capitalization and quoting rules of specific databases
- 45. ><INTRODUCTION TO APACHE CALCITE 45 SqlNode • SqlDialect indicates the capitalization and quoting rules of specific databases
- 46. QUERY OPTIMIZER next 46
- 47. ><INTRODUCTION TO APACHE CALCITE 47 Query Plans • Query plans represent the steps necessary to execute a query
- 48. ><INTRODUCTION TO APACHE CALCITE 48 Query Plans • Query plans represent the steps necessary to execute a query
- 49. ><INTRODUCTION TO APACHE CALCITE 49 Query Plans • Query plans represent the steps necessary to execute a query table scan table scan
- 50. ><INTRODUCTION TO APACHE CALCITE 50 Query Plans • Query plans represent the steps necessary to execute a query inner join table scan table scan
- 51. ><INTRODUCTION TO APACHE CALCITE 51 Query Plans • Query plans represent the steps necessary to execute a query filter inner join table scan table scan
- 52. ><INTRODUCTION TO APACHE CALCITE 52 Query Plans • Query plans represent the steps necessary to execute a query filter inner join project table scan table scan
- 53. ><INTRODUCTION TO APACHE CALCITE 53 Query Plans • Query plans represent the steps necessary to execute a query filter inner join project table scan table scan
- 54. ><INTRODUCTION TO APACHE CALCITE 54 Query Optimization • Optimize logical plan • Goal is typically to try to reduce the amount of data that must be processed early in the plan • Convert logical plan into a physical plan • Physical plan is engine specific and represents the physical execution stages
- 55. ><INTRODUCTION TO APACHE CALCITE 55 Query Optimization • Prune unused fields • Merge projections • Convert subqueries to joins • Reorder joins • Push down projections • Push down filters
- 56. ><INTRODUCTION TO APACHE CALCITE 56 Query Optimization
- 57. ><INTRODUCTION TO APACHE CALCITE 57 Query Optimization
- 58. ><INTRODUCTION TO APACHE CALCITE 58 Query Optimization
- 59. ><INTRODUCTION TO APACHE CALCITE 59 Query Optimization push down project
- 60. ><INTRODUCTION TO APACHE CALCITE 60 Query Optimization push down project push down filter
- 61. ><INTRODUCTION TO APACHE CALCITE 61 Query Optimization
- 62. ><INTRODUCTION TO APACHE CALCITE 62 Key Concepts Relational algebra Row expressions Traits Conventions Rules Planners Programs
- 63. ><INTRODUCTION TO APACHE CALCITE 63 Key Concepts Relational algebra Row expressions Traits Conventions Rules Planners Programs RelNode RexNode RelTrait Convention RelOptRule RelOptPlanner Program
- 64. ><INTRODUCTION TO APACHE CALCITE 64 Relational Algebra • RelNode represents a relational expression • Largely equivalent to Spark’s DataFrame methods • Logical algebra • Physical algebra
- 65. ><INTRODUCTION TO APACHE CALCITE 65 Relational Algebra TableScan Project Filter Aggregate Join Union Intersect Sort
- 66. ><INTRODUCTION TO APACHE CALCITE 66 Relational Algebra TableScan Project Filter Aggregate Join Union Intersect Sort SparkTableScan SparkProject SparkFilter SparkAggregate SparkJoin SparkUnion SparkIntersect SparkSort
- 67. ><INTRODUCTION TO APACHE CALCITE 67 Row Expressions • RexNode represents a row-level expression • Largely equivalent to Spark’s Column functions • Projection fields • Filter condition • Join condition • Sort fields
- 68. ><INTRODUCTION TO APACHE CALCITE 68 Row Expressions Input column ref Literal Struct field access Function call Window expression
- 69. ><INTRODUCTION TO APACHE CALCITE 69 Row Expressions Input column ref Literal Struct field access Function call Window expression RexInputRef RexLiteral RexFieldAccess RexCall RexOver
- 70. ><INTRODUCTION TO APACHE CALCITE 70 Row Expressions
- 71. ><INTRODUCTION TO APACHE CALCITE 71 Row Expressions input ref
- 72. ><INTRODUCTION TO APACHE CALCITE 72 Row Expressions input ref function call
- 73. ><INTRODUCTION TO APACHE CALCITE 73 Traits • Defined by the RelTrait interface • Represent a trait of a relational expression that does not alter execution • Traits are used to validate plan output • Three primary trait types: • Convention • RelCollation • RelDistribution
- 74. ><INTRODUCTION TO APACHE CALCITE 74 Conventions • Convention is a type of RelTrait • A Convention is associated with a RelNode interface • SparkConvention, JdbcConvention, EnumerableConvention, etc • Conventions are used to represent a single data source • Inputs to a relational expression must be in the same convention
- 75. ><INTRODUCTION TO APACHE CALCITE 75 Conventions
- 76. ><INTRODUCTION TO APACHE CALCITE 76 Conventions Spark convention
- 77. ><INTRODUCTION TO APACHE CALCITE 77 Conventions Spark convention JDBC convention
- 78. ><INTRODUCTION TO APACHE CALCITE 78 Conventions Spark convention JDBC convention converter
- 79. ><INTRODUCTION TO APACHE CALCITE 79 Rules • Rules are used to modify query plans • Defined by the RelOptRule interface • Two types of rules: converters and transformers • Converter rules implement Converter and convert from one convention to another • Rules are matched to elements of a query plan using pattern matching • onMatch is called for matched rules • Converter rules applied via convert
- 80. ><INTRODUCTION TO APACHE CALCITE 80 Converter Rule
- 81. ><INTRODUCTION TO APACHE CALCITE 81 Converter Rule expression type
- 82. ><INTRODUCTION TO APACHE CALCITE 82 Converter Rule expression type input convention
- 83. ><INTRODUCTION TO APACHE CALCITE 83 Converter Rule expression type input convention converted convention
- 84. ><INTRODUCTION TO APACHE CALCITE 84 Converter Rule expression type input convention converted convention converter function
- 85. ><INTRODUCTION TO APACHE CALCITE 85 Pattern Matching
- 86. ><INTRODUCTION TO APACHE CALCITE 86 Pattern Matching
- 87. ><INTRODUCTION TO APACHE CALCITE 87 Pattern Matching no match :-(
- 88. ><INTRODUCTION TO APACHE CALCITE 88 Pattern Matching no match :-(
- 89. ><INTRODUCTION TO APACHE CALCITE 89 Pattern Matching match! no match :-(
- 90. ><INTRODUCTION TO APACHE CALCITE 90 Planners • Planners implement the RelOptPlanner interface • Two types of planners: • HepPlanner • VolcanoPlanner
- 91. ><INTRODUCTION TO APACHE CALCITE 91 Heuristic Optimization • HepPlanner is a heuristic optimizer similar to Spark’s optimizer • Applies all matching rules until none can be applied • Heuristic optimization is faster than cost- based optimization • Risk of infinite recursion if rules make opposing changes to the plan
- 92. ><INTRODUCTION TO APACHE CALCITE 92 Cost-based Optimization • VolcanoPlanner is a cost-based optimizer • Applies matching rules iteratively, selecting the plan with the cheapest cost on each iteration • Costs are provided by relational expressions • Not all possible plans can be computed • Stops optimization when the cost does not significantly improve through a determinable number of iterations
- 93. ><INTRODUCTION TO APACHE CALCITE 93 Cost-based Optimization • Cost is provided by each RelNode • Cost is represented by RelOptCost • Cost typically includes row count, I/O, and CPU cost • Cost estimates are relative • Statistics are used to improve accuracy of cost estimations • Calcite provides utilities for computing various resource-related statistics for use in cost estimations
- 94. ><INTRODUCTION TO APACHE CALCITE 94 Cost-based Optimization
- 95. ><INTRODUCTION TO APACHE CALCITE 95 Cost-based Optimization
- 96. ><INTRODUCTION TO APACHE CALCITE 96 Cost-based Optimization
- 97. ><INTRODUCTION TO APACHE CALCITE 97 Cost-based Optimization
- 98. ><INTRODUCTION TO APACHE CALCITE 98 Cost-based Optimization
- 99. PUTTING IT ALL TOGETHER next 99
- 100. ><INTRODUCTION TO APACHE CALCITE 100 Putting it all together
- 101. ><INTRODUCTION TO APACHE CALCITE 101 Putting it all together
- 102. ><INTRODUCTION TO APACHE CALCITE 102 Putting it all together
- 103. ><INTRODUCTION TO APACHE CALCITE 103 Putting it all together
- 104. ><INTRODUCTION TO APACHE CALCITE 104 Putting it all together
- 105. ><INTRODUCTION TO APACHE CALCITE 105 Putting it all together
- 106. ><INTRODUCTION TO APACHE CALCITE 106 Putting it all together
- 107. ><INTRODUCTION TO APACHE CALCITE 107 Putting it all together
- 108. ><INTRODUCTION TO APACHE CALCITE 108 Putting it all together
- 109. ><INTRODUCTION TO APACHE CALCITE 109 Putting it all together