Assembling a Query Engine From Spare Parts

Assembling a Query Engine From Spare Parts

ABSTRACT

Building a new cloud data warehouse is a daunting challenge, requiring massive investments into both the query engine and surrounding cloud infrastructure. Given the mature space, it might seem like a Herculean task to enter the market as a small startup.
At Firebolt we assembled a working, high-performance cloud data warehouse in less than 18 months. We achieved this by building our query engine on top of existing projects and then investing heavily into differentiating features. This paper presents our decision-making and learned lessons along the way.
构建一个新的云数据仓库是一项艰巨的挑战，需要对查询引擎和周围的云基础设施进行大量投资。考虑到成熟的空间，作为一家小型初创公司进入市场似乎是一项艰巨的任务。
在Firebolt，我们在不到18个月的时间里组装了一个可工作的高性能云数据仓库。我们通过在现有项目的基础上构建查询引擎，然后在差异化功能上投入巨资来实现这一点。本文介绍了我们的决策过程和经验教训。

INTRODUCTION

Firebolt is a modern cloud data warehouse built to support userfacing, data-intensive applications [12]. These workloads are challenging, as users expect queries to return in tens of milliseconds. Additionally, user-facing applications can have thousands of simultaneous users and queries. They exhibit many queries-per-second (QPS), as well as high concurrency. Firebolt是一个现代云数据仓库，旨在支持面向用户的数据密集型应用程序[12]。这些工作负载具有挑战性，因为用户期望查询在几十毫秒内返回。此外，面向用户的应用程序可以同时拥有数千个用户和查询。它们表现出每秒许多查询（QPS）以及高并发性。

Building a database management system is hard, as it consists of multiple complex components. These include a query engine, storage engine, transaction manager, and a system catalog. This challenge is amplified when building a cloud data warehouse, as it adds additional layers of complexity to the system. This includes cloud platform infrastructure, cloud storage management, SaaS components, and much more. This is exemplified in Figure 1, presenting Firebolt’s high-level architecture. 构建一个数据库管理系统是困难的，因为它由多个复杂的组件组成。其中包括查询引擎、存储引擎、事务管理器和系统目录。当构建云数据仓库时，这一挑战会被放大，因为它会给系统增加额外的复杂性。这包括云平台基础设施、云存储管理、SaaS组件等等。图1展示了Firebolt的高级体系结构。

All of these components are necessary even for a barebones system. On top of that, significant engineering effort needs to be invested to build out differentiating features. 所有这些组件都是必需的，即使对于裸骨骼系统也是如此。除此之外，还需要投入大量的工程工作来构建差异化功能。

Even for a large team, building such a system from scratch would take multiple years. At Firebolt, we were able to launch our cloud data warehouse for real customers running production workloads in under 18 months. In order to achieve this, we assembled Firebolt from multiple existing components. While some only required small modifications, others were used as a starting point and ended up changing significantly. 即使对于一个大型团队来说，从头开始构建这样一个系统也需要数年时间。在Firebolt，我们能够在不到18个月的时间内为运行生产工作负载的真实客户推出云数据仓库。为了实现这一点，我们用多个现有组件组装了Firebolt。虽然有些只需要小的修改，但其他的则被用作起点，最终发生了重大变化。

This paper presents how we assembled our high-performance query engine from existing components. It is structured as follows. Section 2 focuses on the decisions made while building out the engine itself. Afterwards, Section 3 describes how we leverage open-source tools in order to continuously test our query engine. We summarize our lessons learned in Section 4 and conclude in Section 5. 本文介绍了我们如何从现有组件组装高性能查询引擎。其结构如下。第2节重点介绍在构建引擎本身时所做的决策。之后，第3节描述了我们如何利用开源工具来持续测试我们的查询引擎。我们在第4节中总结了我们的经验教训，并在第5节中总结。

QUERY ENGINE

This Section outlines the open-source building blocks used to assemble our query engine. The design of Firebolt’s engine follows a textbook separation of components [21]. 本节概述了用于组装我们的查询引擎的开源构建块。Firebolt引擎的设计遵循教科书中的组件分离[21]。

Our SQL parser accepts Firebolt’s SQL dialect and converts the user query into an abstract syntax tree (AST). The logical planner then transforms this AST into a logical query plan (LQP). To achieve this, it uses logical metadata such as table and view definitions, data types, and the function catalog. The planner then applies logical transformations to produce an optimized LQP. Sections 2.1 and 2.2 present how we chose an existing project as a foundation for these components at Firebolt.
我们的SQL解析器接受Firebolt的SQL方言，并将用户查询转换为抽象语法树（AST）。然后，逻辑规划器将此AST转换为逻辑查询计划（LQP）。为了实现这一点，它使用了逻辑元数据，如表和视图定义、数据类型和函数目录。规划器然后应用逻辑变换来产生优化的LQP。第2.1节和第2.2节介绍了我们如何选择现有项目作为Firebolt这些组件的基础。

Afterwards, Firebolt’s physical planner constructs a distributed query plan (DQP) from the LQP. To achieve this, it uses physical metadata such as the existence of indexes, table cardinalities, and data distribution. Our distributed runtime orchestrates the execution of the DQP across a cluster of Firebolt nodes. The responsibilities include scheduling, data exchange between stages, and query fault tolerance. Finally, the local runtime executes relational operators within a single Firebolt node. Section 2.3 outlines how we decided on an existing project to lay the foundations for our runtime.
然后，Firebolt的物理规划器从LQP构建一个分布式查询计划（DQP）。为了实现这一点，它使用了物理元数据，如索引的存在、表基数和数据分布。我们的分布式运行时在Firebolt节点集群中协调DQP的执行。职责包括调度、阶段之间的数据交换和查询容错。最后，本地运行时在单个Firebolt节点内执行关系运算符。第2.3节概述了我们如何决定现有项目，为我们的运行时间奠定基础。

We also present some of the engineering challenges encountered when basing the planner and runtime on existing projects. Section 2.4 shows how communication between the planner and runtime evolved over time. Our push towards a custom-built distributed runtime is outlined in Section 2.5. 我们还介绍了在基于现有项目的计划器和运行时遇到的一些工程挑战。第2.4节显示了计划器和运行时之间的通信是如何随着时间的推移而演变的。第2.5节概述了我们对定制分布式运行时的推动。

SQL Dialect

Cloud data warehouses do not exist in a vacuum – they are part of a wider data ecosystem. This ecosystem encompasses tools for ETL/ELT, BI, reporting, data science, ML and data observability. Examples include Fivetran, Dbt, Tableau, Looker and Monte-Carlo. This is outlined in Figure 2. As users build their applications on top of these tools, having extensive integrations with the wider ecosystem is critical to Firebolt’s success. After all, nobody wants a cloud data warehouse with no tools that can talk to it! 云数据仓库并不存在于真空中——它们是更广泛的数据生态系统的一部分。这个生态系统包括ETL/ELT、BI、报告、数据科学、ML和数据可观察性的工具。示例包括Fivetran、Dbt、Tableau、Looker和Monte Carlo。如图2所示。当用户在这些工具之上构建应用程序时，与更广泛的生态系统进行广泛的集成对Firebolt的成功至关重要。毕竟，没有人想要一个没有工具可以与之对话的云数据仓库！

Fortunately, all of the above tools talk to the data warehouse using SQL. This greatly simplifies integrating with the ecosystem. Challenges remain nevertheless. Despite the existence of the ANSI SQL standard, virtually every database features its own SQL dialect. As a result, the above tools require a variety of custom drivers, connectors and adaptors to support different database systems. 幸运的是，上述所有工具都使用SQL与数据仓库对话。这大大简化了与生态系统的集成。然而，挑战依然存在。尽管存在ANSI SQL标准，但实际上每个数据库都有自己的SQL方言。因此，上述工具需要各种自定义驱动程序、连接器和适配器来支持不同的数据库系统。

To be a successful startup within the cloud data warehousing space and to satisfy customers, integrating with the ecosystem from day one is essential. As a small startup, Firebolt cannot expect large companies behind ecosystem tools to spend time and resources dedicated to adding support through custom connectors and drivers. 要想在云数据仓库领域成为一家成功的初创公司，并让客户满意，从第一天起就与生态系统集成至关重要。作为一家小型初创公司，Firebolt不能指望生态系统工具背后的大公司花费时间和资源专门通过自定义连接器和驱动程序添加支持。

To ease ecosystem integration, we decided that the Firebolt SQL dialect should be similar to an existing, widely adopted SQL dialect. Choosing the Postgres dialect as the north star was an easy choice. It is hugely popular and highly compliant with standard ANSI SQL. Almost every tool in the data stack supports Postgres SQL as a result. 为了简化生态系统集成，我们决定Firebolt SQL方言应该类似于现有的、广泛采用的SQL方言。选择Postgres方言作为北极星是一个容易的选择。它非常流行，并且高度符合标准ANSI SQL。因此，数据堆栈中几乎每个工具都支持PostgresSQL。

It is important to note that being compatible with Postgres SQL does not necessitate being compatible with the Postgres wire protocol [19]. Our drivers communicate with Firebolt through a custom HTTP-based REST protocol. 需要注意的是，与PostgresSQL兼容并不一定要与Postgres有线协议兼容[19]。我们的驱动程序通过自定义的基于HTTP的REST协议与Firebolt进行通信。

SQL Parser and Planner

As outlined in the previous section, it was a requirement for Firebolt’s SQL parser to be very similar to Postgres SQL. It has to fully cover DDL, DML, and DCL statements. However, support for DQL (i.e. SELECT) statements is the most important, as they make up the bulk of the workload. 如前一节所述，Firebolt的SQL解析器需要与PostgresSQL非常相似。它必须完全涵盖DDL、DML和DCL语句。然而，对DQL（即SELECT）语句的支持是最重要的，因为它们构成了大部分工作负载。

We wanted to base the logical planner on an existing project meeting a broad set of requirements. The planner needed to support the most important rules in modern data warehousing, such as predicate pushdown and subquery decorrelation. As part of this, the project needed to also have an extensible framework for rulebased transformations. This would allow us to easily add Firebolt specific rules as we scaled the product. 我们希望将逻辑规划器建立在满足广泛需求的现有项目的基础上。规划者需要支持现代数据仓库中最重要的规则，如谓词下推和子查询去相关。作为其中的一部分，该项目还需要有一个用于基于规则的转换的可扩展框架。这将使我们能够在扩展产品时轻松添加Firebolt特定规则。

In addition to rule-based transformations, the planner needed to support cost-based join reordering. This includes allowing us to build custom statistics sources and cost models. This is especially important given the variety of different index types supported by Firebolt. For example, we use sparse primary and secondary indexes for data pruning, as well as specialized indexes for frequently occurring broadcast joins [24]. 除了基于规则的转换之外，规划者还需要支持基于成本的联接重新排序。这包括允许我们构建自定义的统计数据源和成本模型。考虑到Firebolt支持的各种不同索引类型，这一点尤为重要。例如，我们使用稀疏的主索引和辅助索引进行数据修剪，并使用专用索引进行频繁出现的广播联接[24]。

Finally, the planner needed to support composite data types such as arrays and row (struct) types. These are popular for the user-facing, data-intensive applications targeted by Firebolt. 最后，规划者需要支持复合数据类型，如数组和行（结构）类型。这些对于Firebolt针对的面向用户、数据密集型应用程序来说很受欢迎。

It is possible to pick different projects as the baseline for the parser and planner, respectively. However, these two components share a very complex interface: the AST of a query leaving the parser and entering the planner for semantic analysis. We decided to give preference to projects that include both a parser and a planner. 可以选择不同的项目分别作为解析器和计划器的基线。然而，这两个组件共享一个非常复杂的接口：查询的AST离开解析器，进入计划器进行语义分析。我们决定优先选择同时包含解析器和计划器的项目。

Luckily, there are a variety of open-source projects that we were able to consider as a basis for Firebolt’s SQL parser and planner. These are outlined in the following. 幸运的是，我们可以考虑将各种开源项目作为Firebolt的SQL解析器和计划器的基础。这些概述如下。

Postgres Parser. Using the Postgres parser directly would have been the obvious choice given our desire to be compliant with Postgres SQL. This approach has been used successfully in multiple other systems [8, 20]. Isolating the parser from the Postgres code has been done by the libpg_query project1. It packages the original C-based Postgres parser in a library [13].
Postgres Parser。考虑到我们希望与PostgresSQL兼容，直接使用Postgres解析器将是显而易见的选择。这种方法已在多个其他系统中成功使用[8，20]。libpg_query project1已经将解析器与Postgres代码隔离开来。它将原始的基于C的Postgres解析器封装在一个库中[13]。

While this makes it reasonably easy to build a system on top of the Postgres parser, it is difficult to isolate the planner code without bringing in the rest of the large Postgres codebase. For us, this meant that while going with Postgres would require almost no investment in the parsing layer, it would require significant effort to build a production-grade planner that meets our needs. 虽然这使得在Postgres解析器之上构建系统变得相当容易，但如果不引入大型Postgres代码库的其余部分，则很难隔离计划器代码。对我们来说，这意味着使用Postgres几乎不需要在解析层进行投资，但需要付出大量努力来构建一个满足我们需求的生产级计划器。

ZetaSQL. ZetaSQL is a parser and analyzer from Google2 built in C++. It is an open-source port of GoogleSQL [23]. GoogleSQL powers the cloud products BigQuery [17], Spanner [3] and Dataflow, as well as the Google internal products Dremel, F1 [22], and Procella[7]. It is a cleanly built, extensively tested, and production-ready system.
ZetaSQL是一个基于C++的Google2的解析器和分析器。它是GoogleSQL的一个开源端口[23]。GoogleSQL为云产品BigQuery[17]、Spanner[3]和Dataflow以及谷歌内部产品Dremel、F1[22]和Procella[7]提供支持。它是一个构建干净、经过广泛测试并可用于生产的系统。
However, ZetaSQL provides an opinionated dialect that disagrees with Postgres SQL in many basic features. Additionally, ZetaSQL only supports rudimentary transformations and no feature-rich planner. 然而，ZetaSQL提供了一种固执己见的方言，在许多基本特性上与PostgresSQL不一致。此外，ZetaSQL只支持基本的转换，而不支持功能丰富的计划器。

Calcite. Apache Calcite is a framework that provides query processing, optimization, and query language support for data processing systems [4]. It includes parsers for multiple SQL dialects and a modular and extensible query planner with support for pluggable rules. Calcite is well-built and battle tested. It is used in many high-profile open-source systems such as Apache Hive, Apache Storm, Apache Flink, Druid, and MapD3.
Apache Calcite是一个为数据处理系统提供查询处理、优化和查询语言支持的框架[4]。它包括用于多种SQL方言的解析器，以及一个支持可插入规则的模块化和可扩展的查询规划器。方解石建造精良，久经沙场。它被用于许多知名的开源系统，如Apache Hive、Apache Storm、Apache Flink、Druid和MapD3。
Compared to the other alternatives we considered that are written in C++, Calcite is implemented in Java. 与我们考虑的其他用C++编写的替代方案相比，Calcite是用Java实现的。

DuckDB. DuckDB is an in-memory, in-process analytical database system originating from CWI [20]. It is extensively tested and widely used for interactive data analysis. DuckDB’s query planner supports both rule-based optimizations and cost-based join reordering. DuckDB uses the libpg_query project as a baseline for their parser, providing Postgres SQL complicance. Nowadays, DuckDB ported its parser over to C++ 4.
DuckDB。DuckDB是一个源自CWI[20]的内存中、进程内分析数据库系统。它被广泛测试并广泛用于交互式数据分析。DuckDB的查询规划器支持基于规则的优化和基于成本的联接重新排序。DuckDB使用libpg_query项目作为其解析器的基线，从而提供PostgresSQL兼容性。如今，DuckDB将其解析器移植到了C++4上。
At the time we decided on a project to base our parser and planner on, DuckDB was significantly less mature than it is today. 当我们决定以一个项目作为解析器和计划器的基础时，DuckDB明显不如今天成熟。

Hyrise. Hyrise is an in-memory database developed at HPI [11]. It has a relatively simple code base, making it easy to refactor and extend. Similar to DuckDB, it supports rule-based optimizations and cost-based join reordering. However, as an academic project, Hyrise is not battle tested and does not have extensive SQL coverage.
Hyrise是HPI开发的内存数据库[11]。它有一个相对简单的代码库，使得重构和扩展变得容易。与DuckDB类似，它支持基于规则的优化和基于成本的联接重新排序。然而，作为一个学术项目，Hyrise没有经过战斗测试，也没有广泛的SQL覆盖范围。

We decided early on that we wanted planner and runtime to be written in the same programming language. We do not think that this is a hard requirement when building a new system. There are multiple successful systems with JVM-based planners and high-performance runtimes written in C++ [5, 10]. Nevertheless, we believe that the engine being written in one language allows us to have high velocity as a startup. A lot of work in database systems happens at the intersection of planner and runtime. This is especially true for Firebolt, as we had to integrate both components originating from different open-source projects. Planner and runtime being written in the same programming language makes it easy for developers to work across the stack with minimal context switching. As we decided on a runtime written in C++, we wanted the planner to follow suit. This left us with a choice of either Hyrise or DuckDB. 我们很早就决定，我们希望计划器和运行时使用相同的编程语言编写。我们不认为这是建立新系统时的硬性要求。有许多成功的系统具有基于JVM的规划者和用C++编写的高性能运行时[5，10]。尽管如此，我们相信用一种语言编写的引擎可以让我们在启动时获得高速度。数据库系统中的许多工作都发生在计划器和运行时的交叉点上。对于Firebolt来说尤其如此，因为我们必须集成来自不同开源项目的两个组件。Planner和运行时是用同一种编程语言编写的，这使得开发人员可以轻松地跨栈工作，只需最少的上下文切换。当我们决定用C++编写运行时，我们希望规划者也能效仿。这让我们只能选择Hyrise还是DuckDB。

We decided to build upon the Hyrise project due to its simplicity and extensibility. While getting Firebolt production ready, the fact that Hyrise is an academic system with limited SQL support proved to be a challenge. For engineers building a new database system today, using DuckDB is most likely a better starting point. This is because DuckDB has matured significantly since we started building Firebolt and is widely used nowadays. 由于Hyrise项目的简单性和可扩展性，我们决定在其基础上进行构建。在准备Firebolt产品的同时，Hyrise是一个SQL支持有限的学术系统，这一事实证明是一个挑战。对于今天构建新数据库系统的工程师来说，使用DuckDB很可能是一个更好的起点。这是因为自从我们开始构建Firebolt以来，DuckDB已经显著成熟，并且现在被广泛使用。

At the same time, using Hyrise did prove to be a good choice for our use case. Indeed, our initial instinct to go for it given its relative simplicity turned out to be true. We invested heavily to make Firebolt’s parsing and planning layer production ready. Nowadays, its open-source roots in Hyrise are hard to recognize. Our additions include wide ranging extensions to the parser to give good SQL coverage and considerable changes to the planning layer. Among other things, we have added extensive support for composite data types, changed the representation of the logical query plan to make it easier to build rule-based optimizations and added a variety of new rules to the planner. 同时，对于我们的用例来说，使用Hyrise确实是一个不错的选择。事实上，鉴于它相对简单，我们最初选择它的本能是正确的。我们投入巨资使Firebolt的解析和规划层生产做好准备。如今，它在海瑞的开源根源很难被认可。我们添加了对解析器的广泛扩展，以提供良好的SQL覆盖范围，并对规划层进行了大量更改。除其他外，我们添加了对复合数据类型的广泛支持，更改了逻辑查询计划的表示形式，使构建基于规则的优化变得更容易，并向计划器添加了各种新规则。

Our redesign of the logical query plan was heavily inspired by Calcite. As it is the gold standard of extensible and feature-rich open-source query planners, we decided to utilize many of the concepts found in Calcite’s relational algebra representation. This allows our planner to be future proof and extensible in terms of new SQL features and optimization capabilities. 我们对逻辑查询计划的重新设计在很大程度上受到了Calcite的启发。由于它是可扩展和功能丰富的开源查询规划器的黄金标准，我们决定利用Calcite的关系代数表示中的许多概念。这使我们的计划器能够在新的SQL功能和优化功能方面经得起未来考验并具有可扩展性。

Runtime

The runtime is at the heart of a query engine. It is responsible for the query evaluation and has to implement data types, functions and relational operators such as joins and aggregations. 运行时是查询引擎的核心。它负责查询评估，并必须实现数据类型、函数和关系运算符，如联接和聚合。

In a similar vein to the parsing and planning layers, Firebolt had the choice of building a new query engine from scratch or bootstrapping one from an existing open-source project. Many database companies decided to build their runtime from scratch. Examples include CockroachDB [2], Databricks [5] and Snowflake [10] . We believe that in order to disrupt the data warehousing space as a small startup, it is a better choice to start with an existing codebase and allocate our comparatively limited engineering resources to Firebolt’s unique differentiating features. 与解析和规划层类似，Firebolt可以选择从头开始构建一个新的查询引擎，也可以从现有的开源项目中启动一个。许多数据库公司决定从头开始构建运行时。示例包括蟑螂数据库[2]、Databricks[5]和雪花[10]。我们相信，作为一家小型初创公司，为了扰乱数据仓库空间，从现有的代码库开始，并将我们相对有限的工程资源分配给Firebolt独特的差异化功能是一个更好的选择。

We had a few basic guardrails when deciding on which project to use as a baseline for our runtime. 在决定使用哪个项目作为运行时的基线时，我们有一些基本的防护措施。

To support user-facing, data-intensive applications, we needed a high-performance query engine allowing for low-latency query processing. There are two modern approaches to building high performance runtimes – vectorization [6] and code generation [18]. We decided that we wanted to build Firebolt on top of a vectorized runtime. While building a low-latency code generating engine is possible, it requires substantial investment into an advanced compilation stack [15, 16]. This increases engine complexity, makes it harder to onboard new engineers, and retain high development velocity. At the same time, code generating and vectorized engines often perform similarly for OLAP workloads [14]. 为了支持面向用户的数据密集型应用程序，我们需要一个允许低延迟查询处理的高性能查询引擎。构建高性能运行时有两种现代方法——矢量化[6]和代码生成[18]。我们决定在矢量化运行时之上构建Firebolt。虽然构建低延迟代码生成引擎是可能的，但它需要对高级编译堆栈进行大量投资[15，16]。这增加了发动机的复杂性，使新工程师更难加入，并保持了高的开发速度。同时，代码生成和矢量化引擎通常对OLAP工作负载执行类似的操作[14]。

We also wanted the engine to be robust and have basic support for distributed data processing. While there are a variety of interesting academic and experimental open-source query engines, many of those are not production ready or do not support distributed query execution. Starting out with a battle-tested and horizontally scalable engine allowed us to quickly assemble a robust query engine that could process massive data sets. 我们还希望引擎健壮，并对分布式数据处理提供基本支持。虽然有各种有趣的学术和实验性开源查询引擎，但其中许多还没有准备好生产，或者不支持分布式查询执行。从一个经过战斗测试和水平可扩展的引擎开始，我们可以快速组装一个强大的查询引擎，可以处理大量数据集。

Alongside our runtime, we also wanted to bootstrap our storage engine, in particular the file format. In order to build a high performance query engine, it was essential that the storage engine uses a columnar data layout to efficiently support OLAP workloads [1]. The challenges when bootstrapping a query runtime and storage engine are similar to those encountered when bootstrapping a parser and planner. It is possible to pick different systems as a baseline for the storage engine and the query runtime, respectively. However, both components share complex interfaces in order to transfer data between the layers and facilitate data pruning. As such, choosing a single project to supply both components makes it significantly easier to build a high-performance system. 除了运行时，我们还希望引导存储引擎，特别是文件格式。为了构建高性能查询引擎，存储引擎必须使用柱状数据布局来有效支持OLAP工作负载[1]。引导查询运行时和存储引擎时遇到的挑战与引导解析器和计划器时遇到的类似。可以选择不同的系统分别作为存储引擎和查询运行时的基线。然而，这两个组件共享复杂的接口，以便在层之间传输数据并便于数据修剪。因此，选择一个单独的项目来提供两个组件会使构建高性能系统变得更加容易。

While there are multiple projects to choose as a baseline for the SQL parser and planner, there are fewer options when choosing a high-performance, distributed runtime. We quickly narrowed down our choice to ClickHouse [9]. 虽然有多个项目可供选择作为SQL解析器和计划器的基线，但在选择高性能分布式运行时时，选项较少。我们很快将选择范围缩小到ClickHouse[9]。

ClickHouse is designed to be fast5 , and these claims are backed up by benchmarks6. ClickHouse uses vectorized query execution, with limited support for runtime code generation through the LLVM. ClickHouse is battle tested and widely used in production environments 7. Finally, ClickHouse also has its own columnar file format called MergeTree8. MergeTree is tightly integrated with the query runtime and allows for efficient data pruning. All of the above reasons made it easy to choose ClickHouse as a foundation for the Firebolt runtime. ClickHouse被设计为快速5，这些说法得到了基准6的支持。ClickHouse使用矢量化查询执行，对通过LLVM生成运行时代码的支持有限。ClickHouse经过实战测试，并在生产环境中广泛使用7。最后，ClickHouse还有自己的列式文件格式MergeTree8。MergeTree与查询运行时紧密集成，允许高效的数据修剪。以上所有原因使得选择ClickHouse作为Firebolt运行时的基础变得容易。

Connecting Planner and Runtime

Section 2.2 presented our decision to use Hyrise as a basis for our query planner. Afterwards, Section 2.3 showed why we chose to base the Firebolt runtime on ClickHouse. This Section outlines how our approach to communication between planner and runtime evolved over time. 第2.2节介绍了我们使用Hyrise作为查询计划器基础的决定。之后，第2.3节展示了我们为什么选择将Firebolt运行时建立在ClickHouse上。本节概述了我们在计划器和运行时之间的通信方法是如何随着时间的推移而发展的。

Every node in the Firebolt cluster can serve both as query coordinator running parser and planner, and as a runtime worker executing parts of the larger query plan. This is shown in Figure 1. When a query enters the system, it is routed to one of the nodes. Firebolt集群中的每个节点都可以作为运行解析器和计划器的查询协调器，也可以作为执行更大查询计划部分的运行时工作者。如图1所示。当查询进入系统时，它会被路由到其中一个节点。

This node then acts as the coordinator. After query parsing and planning, it needs to initiate query execution. To achieve this, it needs to transform the planner’s optimized LQP into a representation that the ClickHouse-based runtime can understand. The ClickHouse SQL dialect can serve as such a representation. The ClickHouse parser transforms ClickHouse SQL into an internal parse tree that can be executed directly. We used this to quickly build an initial version of cross-component communication. Through a process we called “backtranslate”, an LQP within the planner was transformed back to a representation in ClickHouse SQL. This representation then yielded a ClickHouse query plan that conformed to the optimized LQP chosen by the Firebolt planner. 然后，该节点充当协调器。在查询解析和规划之后，它需要启动查询执行。为了实现这一点，它需要将规划者的优化LQP转换为基于ClickHouse的运行时能够理解的表示。ClickHouse SQL方言可以作为这样一种表示。ClickHouse解析器将ClickHouse SQL转换为可以直接执行的内部解析树。我们使用它来快速构建跨组件通信的初始版本。通过我们称之为“反翻译”的过程，规划者中的LQP被转换回ClickHouse SQL中的表示。然后，这个表示产生了一个ClickHouse查询计划，该计划符合Firebolt计划器选择的优化LQP。

This approach allowed us to get to a working product quickly, but had multiple problems. When connecting planner and runtime this way, a lot of time was spent generating ClickHouse SQL, just to be instantly consumed by the ClickHouse parser again. Even more importantly, a lot of valuable context about the structure of the LQP was lost when going back to a SQL-based representation. 这种方法使我们能够快速获得一个有效的产品，但存在多个问题。当以这种方式连接计划器和运行时时，生成ClickHouse SQL花费了大量时间，只是为了立即再次被ClickHouse解析器消耗掉。更重要的是，当回到基于SQL的表示时，关于LQP结构的许多有价值的上下文都丢失了。

Because of this, we decided to completely replace the backtranslate flow. Nowadays, the Firebolt LQP gets transformed to a distributed query plan directly. This distributed query plan is then disassembled into multiple stages. The coordinator sends stages to different workers within the Firebolt cluster to facilitate distributed query execution. For cross-network communication, we use a custom, protobuf-based serialization format. On the worker nodes, this serialization format is used to assemble the runtime’s vectorized relational operators. In the future, Substrait9 might be a viable alternative to using a custom serialization format. At the time of writing however, Substrait is still undergoing rapid development and is subject to breaking changes. 正因为如此，我们决定完全取代反翻译流。***目前，Firebolt LQP直接转换为分布式查询计划。然后将这个分布式查询计划分解为多个阶段。协调器将阶段发送给Firebolt集群中的不同工作程序，以促进分布式查询执行。***对于跨网络通信，我们使用自定义的、基于protobuf的序列化格式。在工作节点上，这种序列化格式用于组装运行时的矢量化关系运算符。在未来，Subslait9可能是使用自定义序列化格式的可行替代方案。然而，在撰写本文时，Substrait仍在快速发展，并发生了突破性的变化。

Distributed Execution

After choosing ClickHouse as the basis for the Firebolt runtime, we also initially utilized ClickHouse’s approach to distributed query processing. 在选择ClickHouse作为Firebolt运行时的基础之后，我们最初还使用了ClickHouse的分布式查询处理方法。

Distributed query processing in ClickHouse works very well for certain shapes of queries. Examples are queries with selective table scans, distributed aggregations on low-cardinality group-by fields, and broadcast joins. At the same time, ClickHouse does not support many important SQL patterns commonly found in data warehousing such as joins between two large relations, aggregations with high-cardinality group-by fields, window functions without granular PARTITION BY clauses, and large distributed sorts. ClickHouse中的分布式查询处理对于某些形状的查询非常有效。例如，具有选择性表扫描的查询、按字段分组的低基数分布式聚合以及广播联接。同时，ClickHouse不支持数据仓库中常见的许多重要SQL模式，如两个大型关系之间的联接、按字段分组的基数高的聚合、没有细粒度PARTITION by子句的窗口函数以及大型分布式排序。

We decided to move away completely from the ClickHouse distributed execution stack to better serve the data-intensive workloads of our customers. For this, we implemented a new Firebolt distributed processing stack. The optimized LQP returned by our planner is broken up into stages that are connected through shuffle operators. By scaling out, this allows us to execute queries that our original runtime struggled with. Examples include queries with high-cardinality aggregations or joins of two large tables. We will share more about our distributed processing stack in future publications. 我们决定完全放弃ClickHouse分布式执行堆栈，以更好地为客户的数据密集型工作负载提供服务。为此，我们实现了一个新的Firebolt分布式处理堆栈。我们的规划器返回的优化LQP被分解为通过洗牌算子连接的阶段。通过扩展，这使我们能够执行原始运行时难以执行的查询。示例包括具有高基数聚合或两个大型表的联接的查询。我们将在未来的出版物中分享更多关于我们的分布式处理堆栈的信息。

Implementing this new distributed processing stack was only possible because we decided early on that our runtime should be a hard fork of ClickHouse. This gives us the flexibility to perform significant refactoring of interfaces, change the overall architecture of the system, and build a runtime that is optimized for the needs of our customers. 实现这个新的分布式处理堆栈是可能的，因为我们很早就决定我们的运行时应该是ClickHouse的硬分叉。这使我们能够灵活地对接口进行重大重构，更改系统的整体架构，并构建一个针对客户需求进行优化的运行时。

TESTING

Building software is not just about writing code. It is also about making sure that the code works properly. This is especially important for database systems. Users entrust us with their data and rely on us to produce correct query results. This is not something to be taken lightly. 构建软件不仅仅是编写代码。这也是为了确保代码正常工作。这对于数据库系统来说尤其重要。用户将他们的数据委托给我们，并依靠我们产生正确的查询结果。这是不能掉以轻心的。

While we invested heavily into writing our own test cases, we only hand-crafted a small fraction of the test cases that are available elsewhere. Fortunately, when integrating open-source test cases you do not have to choose between different test suites and frameworks. Rather, you can use multiple different ones to combine their own unique strengths. 虽然我们在编写自己的测试用例方面投入了大量资金，但我们只手工制作了其他地方可用的测试用例的一小部分。幸运的是，在集成开源测试用例时，您不必在不同的测试套件和框架之间进行选择。相反，你可以使用多个不同的来结合他们自己的独特优势。

At Firebolt, we aimed to go for maximum coverage and took test cases from wherever we could. This meant that many test cases did not pass for a variety of reasons. Differences in SQL dialect, differences in features, unspecified behaviour, and much more make it hard to port over test cases from other frameworks and have them “just work”. However, integrating with a diverse set of open-source test suites helped us identify real problems. The passing tests make sure we do not regress in functionality or correctness. 在Firebolt，我们的目标是实现最大覆盖率，并尽可能地获取测试用例。这意味着许多测试用例由于各种原因没有通过。SQL方言的差异、功能的差异、未指定的行为等等都使得从其他框架移植测试用例并使其“正常工作”变得困难。然而，与一组不同的开源测试套件集成有助于我们发现真正的问题。通过测试确保我们不会在功能或正确性方面倒退。

Firebolt Query Verification Framework

Most SQL query test frameworks follow similar patterns – they are based on one or multiple test files outlining a data flow. First off, SQL DDL queries may be used to define schemata. Then, SQL DML queries hydrate tables with data. Finally, the SQL queries comprising the actual test are run. These are mostly SELECT statements. Their results are checked against pre-defined expected results.

We have built a custom test framework, named “PeaceKeeper”, to follow a similar pattern. PeaceKeeper knows how to set up a Firebolt cluster in multiple environments. These can be local runs on a developer machine allowing for fast iteration, CI pipelines, or a powerful distributed SQL cluster in the cloud. PeaceKeeper uses test files with input queries and expected results. We have implemented 2K+ Firebolt specific SQL query test cases.

Clickhouse Functional Tests

Given that our runtime is based off ClickHouse, it was natural to integrate the 30K+ functional tests provided by ClickHouse10 . However, since the ClickHouse SQL dialect is very different from Firebolt’s SQL dialect, these tests do not pass through our entire query pipeline. Rather, they work directly against the runtime. We are deprecating these test cases, since our distributed execution stack outlined in Section 2.5 is not integrated with the CickHouse SQL parsing layers anymore.

Postgres Regression Tests

Since Firebolt strives to be as close to Posgres SQL as possible, it made sense to reuse the Postgres test suite. It has 12K+ tests in its regression suite11. Not all of the Postgres constructs are supported in Firebolt. However, for the ones that Firebolt does support, we are able to verify that Firebolt behaves in the same way as Postgres. This was done through an automated script that converts the Postgres regression tests into the PeaceKeeper format.

ZetaSQL Compliance Tests

ZetaSQL has 60K+ tests in its compliance test suite12. The distinctive thing about the ZetaSQL compliance tests is that a very large portion of the test cases focus on SQL expressions and individual functions, extensively covering different boundary conditions – something that many other test suites only do in a cursory manner. ZetaSQL’s compliance tests are mostly code-based as opposed to file-based. In order to keep our testing tools consistent, we wrote a translator program to capture the test queries in the PeaceKeeper format.

SQLLogicTest

The pinnacle of functional SQL testing is SQLLite’s SQLLogicTest framework13. It contains 7M+ test queries (that’s 7 million!). We again wrote a script that ports the SQLLogicTests into PeaceKeeper format. A similar approach was taken by DuckDB14. Due to the sheer volume of queries, we are still in the process of going over each individual result and sorting out errors.

LESSONS LEARNED

In the following, we want to briefly summarize some of the lessons learned while building Firebolt. 在下文中，我们想简要总结一下在构建Firebolt时学到的一些经验教训。

Choosing a Solid Foundation. Especially for query parsers and planners, a wide variety of mature projects can be used as a baseline. While we found Hyrise to be a great foundation and are happy with the choice we made at the time, we would recommend using DuckDB or Calcite for engineers assembling a new system today. Although there are fewer choices for a high-performance distributed runtime, we found ClickHouse to be a robust and extensible foundation. When building a production system, we recommend choosing battle-tested projects as a starting point.
选择坚实的基础。特别是对于查询解析器和规划者来说，可以使用各种成熟的项目作为基线。虽然我们发现Hyrise是一个很好的基础，并对我们当时的选择感到满意，但我们建议工程师今天使用DuckDB或Calcite来组装新系统。尽管高性能分布式运行时的选择较少，但我们发现ClickHouse是一个强大且可扩展的基础。在构建生产系统时，我们建议选择经过战斗测试的项目作为起点。

Building in a Single Language. Assembling the system from projects written in a single programming language allows for higher velocity. This is especially true in the early days of a startup when developers might have to frequently switch between different components or build features across the whole stack.
用单一语言构建。从用单一编程语言编写的项目中组装系统可以获得更高的速度。在创业初期尤其如此，因为开发人员可能不得不频繁地在不同组件之间切换，或者在整个堆栈中构建功能。

Connecting Different Systems. When choosing different systems for e.g. the planner and runtime, engineering teams need to invest heavily in building clean interfaces between these components. We expect similar effects when basing the parser and planner, or the runtime and storage engine on different projects. Because of this,we recommend choosing as few systems as possible in order to assemble a new database management system. We only chose two systems written in the same programming language at Firebolt. The effort required to connect them and turn them into a production-ready system was still significant. Moving the systems ever closer requires a concerted effort and is still ongoing. Our new distributed execution stack outlined in Section 2.5 is an example for this. But much work remains to be done. A good example is unifying the type systems across planner and runtime. We believe that not having to integrate widely different projects is the main benefit of building a new system from scratch.
连接不同的系统。在为规划器和运行时等选择不同的系统时，工程团队需要投入大量资金在这些组件之间构建干净的接口。当解析器和计划器，或者运行时和存储引擎基于不同的项目时，我们预计会产生类似的效果。因此，我们建议选择尽可能少的系统，以便组装一个新的数据库管理系统。在Firebolt，我们只选择了两个用相同编程语言编写的系统。将它们连接起来并将其转变为一个可供生产的系统所需的努力仍然很大。让这些系统越来越近需要协同努力，而且仍在进行中。我们在第2.5节中概述的新分布式执行堆栈就是一个例子。但还有许多工作要做。一个很好的例子是跨计划器和运行时统一类型系统。我们认为，不必集成广泛不同的项目是从头开始构建新系统的主要好处。
Maybe somewhat surprisingly, the Apache Arrow15 project did not play a significant role while composing Firebolt’s query engine. Many existing open-source projects use Arrow to get data into and out of the system. At Firebolt, all data transfer happens within the distributed primitives of the runtime and when returning data from the runtime back to the user. As a result, we did not need to integrate with Arrow to e.g. facilitate cross-project data transfer. We expect that assembling database systems from different components will become easier as Arrow and related projects find even wider adoption across open-source projects.
也许有些令人惊讶的是，ApacheArrow15项目在组成Firebolt的查询引擎时并没有发挥重要作用。许多现有的开源项目都使用Arrow将数据进出系统。在Firebolt中，所有数据传输都发生在运行时的分布式基元中，并且在将数据从运行时返回给用户时。因此，我们不需要与Arrow集成，例如促进跨项目数据传输。我们预计，随着Arrow和相关项目在开源项目中得到更广泛的采用，从不同组件组装数据库系统将变得更容易。

Testing the System. We recommend integrating with as many test frameworks as early as possible. While this might seem like a lot of work in the beginning, it pays off in the long run. It allows teams to quickly catch potential compatibility issues in the targeted SQL dialect, as well as subtle differences in the runtime behaviour compared to other systems. However, sorting through test case failures to figure which ones point at underlying issues and which ones are more benign takes a lot of time.
测试系统。我们建议尽早与尽可能多的测试框架集成。虽然这在一开始看起来是一项艰巨的工作，但从长远来看是有回报的。它允许团队快速发现目标SQL方言中潜在的兼容性问题，以及与其他系统相比运行时行为的细微差异。然而，对测试用例失败进行分类，以确定哪些失败指向潜在问题，哪些更温和，需要花费大量时间。

CONCLUSION

Throughout this paper, we have shown how we used different opensource components as stepping stones to assemble the Firebolt query engine. These components are summarized in Table 1. 在本文中，我们展示了如何使用不同的开源组件作为组装Firebolt查询引擎的垫脚石。这些组成部分汇总在表1中。

The large number of high-quality open-source projects have made it possible for us to build Firebolt in less than 18 months. Going this route as a startup is a smart choice. It allows the organization to quickly converge to a working system. Given the number of mature commercial systems in the space, this allows engineering teams to focus on differentiating features. We are thankful to the open source community, and we intend to contribute back wherever possible. 大量高质量的开源项目使我们能够在不到18个月的时间内构建Firebolt。作为一家初创公司，走这条路是一个明智的选择。它使组织能够快速聚合为一个工作系统。考虑到该领域中成熟的商业系统的数量，这使得工程团队能够专注于差异化功能。我们感谢开源社区，我们打算尽可能地做出贡献。

Assembling a system as outlined in this paper is only the very first step. Building a world-class, high-performance database systems is a marathon and not a sprint – the hard work only starts once a first version of the system is running. For us, deciding to be a hard fork of our open-source roots is essential. It allows for the flexibility to completely redesign the system to better meet our customer’s needs. Firebolt is investing heavily into its engineering teams, and large projects are ongoing to improve or rewrite our storage, execution and metadata layers. We look forward to sharing more about these projects in future publications. 组装本文中概述的系统只是第一步。构建一个世界级的高性能数据库系统是一场马拉松，而不是短跑——只有在系统的第一个版本运行后，艰苦的工作才会开始。对我们来说，决定成为开源根源的硬分叉是至关重要的。它允许灵活地完全重新设计系统，以更好地满足客户的需求。Firebolt正在对其工程团队进行大量投资，大型项目正在进行中，以改进或重写我们的存储、执行和元数据层。我们期待着在未来的出版物中分享更多关于这些项目的信息。

ACKNOWLEDGMENTS
We thank the contributors and maintainers of Hyrise and ClickHouse for providing an exceptional foundation for Firebolt to innovate on.

This work reflects significant effort by Firebolt’s engineering teams. It would not have been possible without the exceptional work of current and past Firebolt engineers Ariel Yaroshevich, Eldad Farkash, Daniel Heilper, Jonathan Goldfeld, Cosmin Pop, Daniel Ungur, Rares Mironeasa, Ronen Fishman, Joel Shalit, Izik Orgad, Ernest Zaslavsky, Jonathan Doron, Svetlana Messel, Artem Yasynskyi, Vadim Entov, Ivan Koptiev, Boaz Farkash, Elad Dolev, Ori Brotovski, Michael Dessenov, Vlad Davidchenko, Kfir Yehuda, Shahar Shalev, Galit Koka Elad, Oren Wegner, Leonid Bobovich, Itay Israelov, Viacheslav Zubko, Michael Umansky, Arkadi Kagan, Noam Fisher, Yarin Hananiya, Kevin Marr, Cody Schwarz, Gil Cizer, Andrey Kisel, Roman Rustakov, Oleksandr Dudchenko, Francesco Dondi, Ido Kidron, Yuval Boker, Sasha Solganik, David Boublil, Michal Szadkowski, Dany Azriel, James Hill, Imre Palik, Ilya Yakubenko, Nikolay Mizin, Octavian Zarzu, Michael Kravchenko, David Welch, Lorenz Hübschle-Schneider, Dima Sporov, Nir Levy, Maxim Axelrod, Sebastian Kurella, Loic Reyreaud, Leonard von Merzljak, Arnaud Comet, Brit Myers, Jusufadis Bakamović, Tobias Humig, Tal Zelig, Shira Fried, Peter McConnell, Zhen Li, Misha Shneerson, Michael Moursalinov, Aurash Behbahani, Zuyu Zhang, Gracco Guimarães, Andrés Senac Gonezález, Arsenii Krasikov, Bruhathi Sundarmurthy, Branden Pleines, Giuseppe Mazzotta, Anton Perkov, Asy Ronen, Andrei Burago, Alex Hall, Ron Dubinsky, and Ying Li.

REFERENCES
[1] Daniel Abadi, Peter Boncz, Stavros Harizopoulos, Stratos Idreos, Samuel Madden, et al. 2013. The design and implementation of modern column-oriented database systems. Foundations and Trends in Databases 5, 3 (2013), 197–280.
[2] Subiotto Marques Alfonso and Rafi Shamim. 2019. How We Built a Vectorized Execution Engine. Cockroach Labs. https://www.cockroachlabs.com/blog/how-we-built-a-vectorized-execution-engine/
[3] David F. Bacon, Nathan Bales, Nico Bruno, Brian F. Cooper, Adam Dickinson, Andrew Fikes, Campbell Fraser, Andrey Gubarev, Milind Joshi, Eugene Kogan, Alexander Lloyd, Sergey Melnik, Rajesh Rao, David Shue, Christopher Taylor, Marcel van der Holst, and Dale Woodford. 2017. Spanner: Becoming a SQL System. In Proceedings of the 2017 ACM International Conference on Management of Data (Chicago, Illinois, USA) (SIGMOD ’17). Association for Computing Machinery, New York, NY, USA, 331–343. https://doi.org/10.1145/3035918.3056103
[4] Edmon Begoli, Jesús Camacho-Rodríguez, Julian Hyde, Michael J. Mior, and Daniel Lemire. 2018. Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources. In Proceedings of the 2018 International Conference on Management of Data (Houston, TX, USA) (SIGMOD’18). Association for Computing Machinery, New York, NY, USA, 221–230. https: //doi.org/10.1145/3183713.3190662
[5] Alexander Behm, Shoumik Palkar, Utkarsh Agarwal, Timothy G. Armstrong, David Cashman, Ankur, Dave, Todd Greenstein, Shant Hovsepian, Ryan Johnson, Arvind Sai Krishnan, Paul Leventis, Ala, Luszczak, Prashanth Menon, Mostafa Mokhtar, Sameer Paranjpye, Greg Rahn, Bart Samwel, Tom van Bussel, Herman van Hovell, Maryann Xue, Reynold Xin, and Matei A. Zaharia. 2022. Photon: A Fast Query Engine for Lakehouse Systems. In Proceedings of the 2022 ACM SIGMOD International Conference on Management of Data (New York, New York, USA) (SIGMOD ’22). Association for Computing Machinery, New York, NY, USA, 14. https://doi.org/10.1145/3514221.3526054
[6] Peter A Boncz, Marcin Zukowski, and Niels Nes. 2005. MonetDB/X100: Hyper Pipelining Query Execution… In CIDR, Vol. 5. Citeseer, 225–237.
[7] Biswapesh Chattopadhyay, Priyam Dutta, Weiran Liu, Ott Tinn, Andrew Mccormick, Aniket Mokashi, Paul Harvey, Hector Gonzalez, David Lomax, Sagar Mittal, Roee Ebenstein, Nikita Mikhaylin, Hung-ching Lee, Xiaoyan Zhao, Tony Xu, Luis Perez, Farhad Shahmohammadi, Tran Bui, Neil McKay, Selcuk Aya, Vera Lychagina, and Brett Elliott. 2019. Procella: Unifying Serving and Analytical Data at YouTube. Proceedings of the VLDB Endowment 12, 12 (aug 2019), 2022–2034. https://doi.org/10.14778/3352063.3352121
[8] Sid Choudhury. 2020. Why We Built YugabyteDB by Reusing the PostgreSQL Query
Layer. YugabyteDB. https://blog.yugabyte.com/why-we-built-yugabytedb-byreusing-the-postgresql-query-layer/
[9] ClickHouse. 2021. Overview of ClickHouse Architecture. ClickHouse. https: //clickhouse.com/docs/en/development/architecture/
[10] Benoit Dageville, Thierry Cruanes, Marcin Zukowski, Vadim Antonov, Artin Avanes, Jon Bock, Jonathan Claybaugh, Daniel Engovatov, Martin Hentschel, Jiansheng Huang, Allison W. Lee, Ashish Motivala, Abdul Q. Munir, Steven Pelley, Peter Povinec, Greg Rahn, Spyridon Triantafyllis, and Philipp Unterbrunner. 2016. The Snowflake Elastic Data Warehouse. In Proceedings of the 2016 International Conference on Management of Data (San Francisco, California, USA) (SIGMOD’16). Association for Computing Machinery, New York, NY, USA, 215–226. https: //doi.org/10.1145/2882903.2903741

[11] Markus Dreseler, Jan Kossmann, Martin Boissier, Stefan Klauck, Matthias Uflacker, and Hasso Plattner. 2019. Hyrise Re-engineered: An Extensible Database System for Research in Relational In-Memory Data Management. In Advances in Database Technology - 22nd International Conference on Extending Database Technology, EDBT 2019, Lisbon, Portugal, March 26-29, 2019, Melanie Herschel, Helena Galhardas, Berthold Reinwald, Irini Fundulaki, Carsten Binnig, and Zoi Kaoudi (Eds.). OpenProceedings.org, 313–324. https://doi.org/10.5441/002/edbt.2019.28
[12] Firebolt. 2022. The Firebolt Cloud Data Warehouse Whitepaper. Firebolt Analytics. https://www.firebolt.io/resources/firebolt-cloud-data-warehouse-whitepaper
[13] Lukas Fittl. 2021. Introducing pg_query 2.0: The easiest way to parse Postgres queries. pganalyze. https://pganalyze.com/blog/pg-query-2-0-postgres-query-parser
[14] Timo Kersten, Viktor Leis, Alfons Kemper, Thomas Neumann, Andrew Pavlo, and Peter Boncz. 2018. Everything You Always Wanted to Know about Compiled and Vectorized Queries but Were Afraid to Ask. Proceedings of the VLDB Endowment 11, 13 (sep 2018), 2209–2222.
[15] Timo Kersten, Viktor Leis, and Thomas Neumann. 2021. Tidy Tuples and Flying Start: Fast Compilation and Fast Execution of Relational Queries in Umbra. The VLDB Journal 30, 5 (sep 2021), 883–905. https://doi.org/10.1007/s00778-020- 00643-4
[16] André Kohn, Viktor Leis, and Thomas Neumann. 2018. Adaptive execution of compiled queries. In 2018 IEEE 34th International Conference on Data Engineering (ICDE). IEEE, 197–208. https://doi.org/10.1109/ICDE.2018.00027
[17] Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, Theo Vassilakis, Hossein Ahmadi, Dan Delorey, Slava Min, Mosha Pasumansky, and Jeff Shute. 2020. Dremel: A Decade of Interactive SQL Analysis at Web Scale. Proceedings of the VLDB Endowment 13, 12 (aug 2020), 3461–3472. https://doi.org/10.14778/3415478.3415568
[18] Thomas Neumann. 2011. Efficiently Compiling Efficient Query Plans for Modern Hardware. Proceedings of the VLDB Endowment 4, 9 (jun 2011), 539–550. https: //doi.org/10.14778/2002938.2002940
[19] PostgreSQL. 2021. Frontend/Backend Protocol. Postgres. https://www.postgresql.org/docs/14/protocol.html
[20] Mark Raasveldt and Hannes Mühleisen. 2019. DuckDB: An Embeddable Analytical Database. In Proceedings of the 2019 International Conference on Management of Data (Amsterdam, Netherlands) (SIGMOD ’19). Association for Computing Machinery, New York, NY, USA, 1981–1984. https://doi.org/10.1145/3299869. 3320212

[21] Raghu Ramakrishnan and Johannes Gehrke. 2000. Database Management Systems (2nd ed.). McGraw-Hill, Inc., USA.
[22] Bart Samwel, John Cieslewicz, Ben Handy, Jason Govig, Petros Venetis, Chanjun Yang, Keith Peters, Jeff Shute, Daniel Tenedorio, Himani Apte, Felix Weigel, David Wilhite, Jiacheng Yang, Jun Xu, Jiexing Li, Zhan Yuan, Craig Chasseur, Qiang Zeng, Ian Rae, Anurag Biyani, Andrew Harn, Yang Xia, Andrey Gubichev, Amr El-Helw, Orri Erling, Zhepeng Yan, Mohan Yang, Yiqun Wei, Thanh Do, Colin Zheng, Goetz Graefe, Somayeh Sardashti, Ahmed M. Aly, Divy Agrawal, Ashish Gupta, and Shiv Venkataraman. 2018. F1 Query: Declarative Querying at Scale. Proceedings of the VLDB Endowment 11, 12 (aug 2018), 1835–1848. https://doi.org/10.14778/3229863.3229871
[23] Jeff Shute. 2022. GoogleSQL: A SQL Language as a Component. (2022). https:
//cdmsworkshop.github.io/2022/invited.html#invited5 First International Workshop on Composable Data Management Systems.
[24] Octavian Zarzu. 2023. Firebolt Indexes in Action. Firebolt Analytics. https:
//www.firebolt.io/blog/firebolt-indexes-in-action