Glossary

Apache Parquet is an open-source columnar storage format optimized for large-scale data processing and analytics. It's widely adopted across big data ecosystems, including Apache Hive, Spark, Doris, Trino, Presto, and many others.

Apache Paimon is a high-performance streaming-batch unified table format and data lake storage system designed specifically for real-time data processing. It supports transactions, consistent views, incremental read/write operations, and schema evolution, providing essential capabilities required by modern data lake architectures.

Apache ORC (Optimized Row Columnar) is an open-source columnar storage format optimized for large-scale data storage and analytics. Developed by Hortonworks in 2013, it has become an Apache top-level project and is widely used in big data ecosystems including Apache Hive, Spark, Presto, Trino, and more.

Apache Iceberg is an open-source large-scale analytical table format initiated by Netflix and donated to Apache, designed to address the limitations of traditional Hive table formats in consistency, performance, and metadata management. In today's lakehouse architectures with multi-engine concurrent access and frequent schema evolution, Iceberg provides ACID transactions, hidden partitioning, time travel capabilities, making it highly sought after.

Apache Hudi (Hadoop Upserts Deletes Incrementals) is an open-source data lake platform originally developed by Uber and became an Apache top-level project in 2019. By providing transactional capabilities, incremental processing, and consistency control for data lakes, it transforms traditional data lakes into modern lakehouses. As users increasingly focus on real-time capabilities, update functionality, and cost efficiency in massive data scenarios, Hudi emerges as the solution to address these pain points.

Apache Hive is a distributed, fault-tolerant data warehouse system built on Hadoop that supports reading, writing, and managing massive datasets (typically at petabyte scale) using HiveQL, an SQL-like language. As big data scales continue to grow exponentially, enterprises increasingly demand familiar SQL interfaces for processing enormous datasets. Hive emerged precisely to address this need, delivering tremendous productivity value.

Delta Lake is an open-source storage format that combines Apache Parquet files with a powerful metadata transaction log. It brings ACID transactions, consistency guarantees, and data versioning capabilities to data lakes. As large-scale data lakes have been widely deployed in enterprises, using Parquet alone cannot solve issues like performance bottlenecks, data consistency, and poor governance. Delta Lake emerged to address these challenges and has become an essential foundation for building modern lakehouse architectures.

Lakehouse (Data Lake + Data Warehouse) is a unified architecture that aims to provide data warehouse-level transactional capabilities, management capabilities, and query performance on top of a data lake foundation. It not only retains the low cost and flexibility of data lakes but also provides the consistency and high-performance analytical capabilities of data warehouses.

Apache Doris is an MPP-based real-time data warehouse known for its high query speed. For queries on large datasets, it returns results in sub-seconds. It supports both high-concurrency point queries and high-throughput complex analysis. It can be used for report analysis, ad-hoc queries, unified data warehouse, and data lake query acceleration. Based on Apache Doris, users can build applications for user behavior analysis, A/B testing platform, log analysis, user profile analysis, and e-commerce order analysis.

An analytics database is a specialized database management system optimized for Online Analytical Processing (OLAP), designed to handle complex queries, aggregations, and analytical workloads across large datasets. Unlike traditional transactional databases that focus on operational efficiency and data consistency, analytics databases prioritize query performance, data compression, and support for multidimensional analysis. Modern analytics databases leverage columnar storage, massively parallel processing (MPP) architectures, and vectorized execution engines to deliver sub-second response times on petabyte-scale datasets, making them essential for business intelligence, data science, and real-time decision-making applications.