How Meituan consolidated its analytics stack on Apache Doris

This article is adapted from a talk given by a Meituan technical expert at an Apache Doris online event.

Meituan's business runs on data. Strategy, product iteration, business analysis, and marketing campaigns all depend on data that is accurate and current. For years, serving that analysis meant operating Hadoop, Kylin, Druid, and several other engines side by side, each chosen for a specific workload. This architecture held for a while, but four challenges were compounding underneath it.

The four challenges that drove the search for a new solution

As Meituan scales, the challenges begin to surface, and the most prominent ones are the following:

• The first was latency. Most analytical reports at Meituan need to respond within seconds, and some business-facing scenarios need sub-second returns. That expectation applies regardless of query complexity.

• The second was volume. Core business tables passed hundreds of billions of rows, which is the point at which a traditional warehouse starts having to choose between serving all the data and serving it quickly.

• The third was the range of workloads the platform had to serve. Transactions, operations, users, traffic, and merchants all live on the same platform, and each one makes different demands on query patterns, concurrency, data freshness, and stability. An engine tuned for one is rarely a good fit for the next.

• The fourth was where the platform was heading. As business intelligence (BI) systems across the business lines consolidated, whatever sat underneath them would have to serve unified analytics and absorb the complex queries and elastic compute that AI applications bring.

Underneath all four sat a cost problem. Running several engines meant paying for maintenance several times over, paying for storage several times over, and asking every team to know which engine answered which kind of question.

Taken together, those four defined what a replacement would have to absorb. Two production workloads showed what that meant in practice.

The workloads that defined the requirement

Two production workloads made the requirements concrete, and they required different trade-offs.

• Business development (BD) productivity analysis for in-store dining spans hundreds of dimensions and dozens of aggregate metrics. It also depends on exact deduplication, because the output feeds performance reviews and an approximate count will not do. That combination leans hard on Bitmap dedup, pre-aggregation, and raw query speed.

• Merchant operating reports, on the other hand, serve real-time queries to merchants, mostly high-concurrency point lookups on highly selective fields such as merchant ID. Here the pressure is on latency, concurrency, and availability, all at once, because each query sits in the path of a merchant waiting on a dashboard refresh.

A platform that could satisfy both would need to handle complex analysis, exact deduplication, high-concurrency point lookups, real-time ingest, offline processing, and unified governance across business lines. Meituan went looking for one engine that could do all of it.

Choosing Apache Doris

The team chose Apache Doris for its general-purpose characteristics and its performance across a range of workloads.

The platform that emerged has two paths into Doris. User behavior data, DB data, and log data all collect into Kafka first. From there, the real-time path runs through Flink, which processes the data and writes to Doris to power live dashboards, operational monitoring, and online analysis. The offline path lands in the Hive/HDFS warehouse, where Spark and Hive handle cleaning, aggregation, and modeling before the results sync into Doris for serving.

Figure 1. Meituan's end-to-end data platform, with Doris in the data serving layer.

In the architecture, Doris sits in the data serving layer, taking results from both the real-time path and the offline warehouse and handing them up to business reports, operational analysis, merchant services, and traffic analysis. It carries low-latency queries, high-concurrency access, complex multi-table analysis, exact deduplication, real-time writes, and unified serving of offline output.

Extending Doris to new workloads

Consolidation solved the cost and operational overhead problem. Over the next few years, teams brought Doris workloads it had not been asked to handle before, and each one exposed something that needed building, which prompted the team at Meituan to make improvements and contribute them back to the community.

Improving join performance for food delivery analysis

Operational analysis for food delivery leans heavily on multi-table joins. In a distributed system, a conventional shuffle join moves a lot of data across nodes, and at Meituan's volumes those queries were running past their timeouts.

The team's answer was Colocate Join: distribute and organize the data ahead of time so that rows involved in a join land on the same node wherever possible. The work moves to where the data already is.

Figure 2. Shuffle join versus Colocate Join.

In internal testing, join query performance improved by roughly 3x on average, and the food delivery reports began completing inside their timeout windows.

Figure 3. Join execution time, ShuffleJoin versus ColocatedJoin, by rows scanned per table.

Exact deduplication for traffic and user analysis

Traffic analysis brought a different demand. Daily active user (DAU) calculation, retention analysis, and behavioral analysis all depend on exact deduplication.

Meituan put substantial optimization work into Doris Bitmap, spanning compute throughput, memory handling, pre-aggregation, and parallelism. The change that mattered most was in data distribution, where controlling the cardinality of the data distribution improved overall exact-deduplication performance at high user cardinality.

Figure 4. The Bitmap exact-deduplication optimization areas.

Afterward, a single SQL query could span tens of billions of rows. At cardinalities in the hundreds of millions, metric computation ran 4x to 5x faster on average, with single-table queries returning inside 10 seconds.

Figure 5. DAU metric query latency, before and after bitmap orthogonal bucketing.

Those numbers were what allowed Doris to absorb the exact-dedup analysis that pre-aggregation engines like Kylin had been carrying.

Isolating data production from query serving

The third came from a timing requirement. Some workloads at Meituan have to produce, process, and land data within 10 minutes. Running that processing inside Doris put it in direct competition with online queries for the same resources, and query stability suffered.

The team built a Spark on Doris architecture on top of the Doris Spark Connector, giving the heavy processing its own place to run. Spark does the complex work, then writes finished results into Doris for serving.

Figure 6. ETL flow for the merchandise front end using Spark on Doris.

The pattern paid off in four ways. It enabled cross-cluster reads and writes, which eliminated duplicated computation logic. It moved complex processing onto Spark and took pressure off Doris. It let Doris-to-Doris processing jobs connect through a SQL interface. And it gave data production and online queries separate resources, which is what stabilized both.

Version sprawl across 300+ clusters

Each of those wins brought more teams onto the platform, and the footprint grew accordingly: past 300 clusters, hundreds of thousands of CPU cores, and tens of PB of data, with the largest single cluster holding 10 PB.

At that size, the hard problems stopped being about performance. Business lines had adopted Doris at different times over several years, which left the teams running a mix of Doris versions spanning 0.1 all the way through 2.1. Those versions differ in features, in semantics, and in how you operate them. Every difference made troubleshooting slower, made rolling out a new capability harder, and made governing the fleet centrally close to impossible.

Figure 7. Platform scale: clusters, compute, data volume, and version spread.

Resolving the version sprawl issue meant upgrading several hundred production clusters. The team went through the conventional options and found problems with all of them.

In-place upgrade carried too much risk, the version span was wide enough to require several sequential hops plus metadata structure changes along the way, and a failure partway through could leave a cluster with nowhere to roll back to.

A full reload or backup-and-restore ran into issues with the data volume itself. With single clusters holding PB and in some cases 10 PB, a complete migration is slow, risky, and very hard to do without taking a downtime window.

Dual writes work in some situations, but the application-side rework is expensive, and the resulting pipeline is complicated, which made it impractical to push out uniformly across hundreds of clusters.

With none of the conventional paths viable, the requirement came down to a single constraint: move a PB-scale system onto a new version with as little disruption to the business as possible. Meituan found that path in cross-cluster replication.

Upgrading the clusters with cross-cluster replication

Meituan built its upgrade program on Doris called cross-cluster replication (CCR).

Figure 8. What CCR provides: near real-time incremental sync, full DDL/DML support, minimal source-cluster impact.

CCR captures changes from binlog and replays them. The source cluster records data and metadata changes, the target cluster consumes and replays them continuously, and the two stay in incremental sync.

That turns a one-time migration into a continuous catch-up. The new cluster syncs in the background while the old one keeps serving traffic. Once the lag between them converges to an acceptable range, a DNS switch moves traffic across, and the business barely notices the handoff.

Figure 9. Parallel cluster topology, before and after the domain switch.

In production, the two clusters ran side by side. The new one received binlog through the CCR sink, finished its historical replay, then settled into incremental sync. At cutover, the team used dual domains and dual-run: Kafka, Hive, and other ingest jobs pointed at both clusters at once, which kept the pipelines consistent and preserved a fast rollback path throughout the observation window.

Validation turned out to be the part that mattered most. SQL replay alone could not reproduce real business scenarios, so the team added business-side end-to-end validation to confirm that old and new clusters agreed at the level of business semantics. That check is what brought migration risk to an acceptable level.

Running alongside all of it was a governance effort: metadata bloat, replica counts, malformed metadata structures, inconsistent cluster health. That work was a precondition for the upgrade. Some legacy clusters had replica counts in the tens of millions, severe enough that in extreme cases it could affect frontend (FE) node recovery, and moving those clusters before cleaning them up would have been risky on its own terms.

When the upgrade and the governance work were done, overall query and ingest performance had improved by roughly 20% to 40%, reaching 50% in some extreme cases, with a noticeable gain in stability across the fleet.

The more lasting change is in how the platform moves. Meituan used to trade downtime for upgrades. Now continuous data catch-up drives system evolution, which means the architecture can keep advancing without the business having to stop and wait for it.

What Meituan contributed back to Apache Doris

Most of what Meituan built to solve its own problems ended up in Apache Doris, where anyone can use it.

Colocate Join is the clearest example. It started as an answer to food delivery reports timing out, and it was contributed to the community, where it now helps other users facing the same class of join problem. Anyone who runs a colocated join in Doris today is running code that exists because Meituan's delivery analysts could not get their reports to finish.

The Bitmap exact-deduplication work followed the same path into the same engine every Doris user runs, from the RoaringBitmap and single instruction, multiple data (SIMD) throughput work through the memory allocator and bucketing changes. Spark on Doris was built on the community's Doris Spark Connector rather than as a private integration.

The upgrade program generated a second round of contributions. The team backported CCR to older cluster versions, which is what made it usable on the clusters that most needed migrating. They converted DB-level serial sync into table-level parallel execution, so large fleets could actually move at a reasonable pace. And they fixed a run of cross-version issues surfaced by the migration itself: precision differences, optimizer behavior differences, and performance regressions that only show up when you replay years of production data across a version gap.

What comes next

Meituan is now building a unified query and interface layer that pulls SQL query, operational diagnostics, and data access into one abstraction, giving AI applications a standardized and controllable way to query data. The team is also working on an operational agent that analyzes large queries, data skew, and node anomalies automatically, moving toward automated diagnosis. Building the observation, evaluation, and governance systems that agents themselves need is the open question for the next phase.

Figure 10. Target architecture: an AI-enabled BI application over a query engine interface layer.

Apache Doris is moving in the same direction, adding capabilities for agent-native scenarios: hybrid retrieval and analysis, VARIANT, inverted indexes, sort keys, partition pruning, and caching. Together, they give the traces, events, queries, evaluation results, and multi-tenant observations an agent produces a high-performance store to run against. Litefuse builds on Doris to offer an observability and evaluation platform for agents, making the full agent path traceable and its results measurable. Meituan plans to explore that direction actively.

Join the Apache Doris community on Slack and connect with other Doris experts and users. If you're looking for a fully managed Apache Doris cloud service, contact the VeloDB team.