Querying Druid in SQL with Superset

Editor's Notes

• #4: Druid emerges recently as a is an open-source data store designed for sub-second queries on real-time and historical data. It is usually used to query event data. It can ingest event data in real time and allows flexible data exploration and data aggregation. Right now in Oath, a lot of data sciences want to do a ad-hoc query on users event data and Druid will be their first choice. Now, we will go over how to work with druid when we need to do a adhoc query.
• #5: In this presentation, we will first go over the reason why we want a SQL interface on the top of Druid. In the meantime, we will briefly introduce Druid and other existing solution. After that, we will focus on the improvement needed to be done for the SQL interface. This part will include our contribution on the open source project. At last, we will introduce how we combine the SQL interface with a neat User interface for a wider range of users to work on Druid.
• #6: Let’s say I am a data scientist and I may want to run this SQL on druid. In this SQL query, we are looking for summation of store_sales in California and store_costs in Oregon. The data in October should be excluded. Since now the data is in druid, we will have to write a query that Druid understands. In this case, we have to write a JSON and send it to druid within a http request. So how the json will be like?
• #7: Probably after several experience with druid, ones will be familiar with this format. However, for someone who is already familiar with SQL to translate SQL into this format, it will need some time to learn the druid documentation and probably the syntax of json. It will be great if we can have a SQL interface to query druid data without losing the great performance of druid. Let’s go ahead and explore some SQL interfaces that Druid could work with.
• #9: First option we can have is to use the SQL services provided by Druid. It works similarly as the druid query and we need to send a http request with the SQL statement in a json object. This function is supported by a open source tool called Apache Calcite.
• #10: Need more
• #11: Obviously, Calcite is the core part of the SQL interface on Druid. To develop a SQL interface on Druid, it is necessary to learn about Calcite first. We will briefly introduce Calcite and how it works then go over the contribution we made on Open source Calcite.
• #13: The most important concept in Calcite is the relational algebra. Calcite basically translate SQL statement into a tree structure with nodes representing the logic in it. For example, the tree on the left side is a simple example algebra tree for the SQL statement on the left side. First, the TableScan to lock down the SQL on certain table. Then the filter contains the logic of WHERE value of b column equals to 1. The Project node is not so obvious but in many SQL database the actual column name may contain namespace and the project node will take care these kinds of translation. The Aggregate node includes the logic of summation function we used. The last one is the sort node which corresponds to order by part in SQL statement.
• #14: With the relational logic tree, we now can transform the tree to another tree representing the equivalent logic. After transformation, the new tree should be translated into Druid Query. The transformation rules will subtrees with equivalent logic to be transformed between each other. Like the example here, the first four node can be transformed as a druid groupby query through rules we specified. In the other transformation path, the whole tree can be transformed into Druid topN query node which contains equivalent logic. Now, the question is to pick a certain tree as our final result.
• #15: The final result will be determined by the optimizer in Calcite. In calcite, each node will have cost to quantify the computation power it required to run the logic in the node. Different output trees, therefore, will have different cost. The result will be the final tree with minimum cost. Back to our example, the cost of final tree on the top is 10 + 10 20, but another output tree only have one node with 15 cost. The optimizer will then pick the Druid TopN query node as the final result. Now, the final job is to render the druid json query with the logic we have in the druid query node.
• #16: Now, the final job is to render the druid json query with the logic we have in the druid query node. Basically it is like a json writer that generate the query based on the information we had in the node.
• #17: The whole idea of Calcite is brilliant, but at the time when we try to use it, it still has missing piece that can affect the performance. We the decided to contribute to the open source repository to enhance Calcite. These are two major problems we worked on. First, Calcite in that time did not have support on post-aggregation, so part of the computation in function like AVERAGE will be performed in Calcite instead of Druid. Moving those computation to Druid can save memory and time when running query. The second one is the support of filtered Aggregation. Without the support, calcite may assign druid to query all rows and do the filter in Calcite local machine. This could cause a huge fallback in performance when filtered aggregation is involved.
• #18: To add post aggregation support, we add new rules to merge post aggregation node in supported druid query node, so the final tree can include post aggregation logic. Also, a new renderer is needed to render query with post aggregator. Now, when possible, the post aggregation will be pushed to Druid query node.
• #19: Similar as the post aggregation support, we add new rules to deal with the filter node right before the aggregate node. The general idea here is to move inner filter to outer filter