TL;DR:
A leadind payment platform, handling over 600 million daily security events, replaced its complex Elasticsearch and Hudi-based stack with Apache Doris. This move was driven by the need for higher performance, lower costs, and a simpler architecture for their critical financial security data platform. The results were dramatic:
- Up to 56x faster queries compared to Elasticsearch.
- 50% reduction in storage costs.
- 4x increase in write throughput.
- A simplified, unified architecture that eliminated data consistency issues and boosted developer efficiency.
The Challenge: A Sprawling, Costly Architecture for Critical Security Data
As a major financial infrastructure provider with over 600 million registered users, this payment platform's security is paramount. Their financial security data platform is the first line of defense, processing over 600 million security logs and events daily to fend off millions of network attacks.
However, their data architecture struggled to keep pace. Their journey began with Architecture 1.0, using Elasticsearch for log storage. As data volumes soared, this approach hit a wall:
- High Storage Costs: With a compression ratio of only ~1.5x, storing massive datasets was prohibitively expensive.
- Weak Analytics: Elasticsearch lacked support for complex analytics like multi-table JOINs, hindering deep security analysis.
- Performance Bottlenecks: Complex queries or those spanning more than 20 days could easily crash the cluster.
To address the analytics limitations, they evolved to Architecture 2.0, introducing Hudi, Hive, and Presto. While this improved interactive analysis, it created a new set of problems:
- Excessive Complexity: The multi-component stack was a nightmare to operate and maintain.
- Data Consistency Challenges: Keeping data synchronized across Flink, Hudi, Elasticsearch, and Presto was a constant, error-prone challenge.
- High Resource Consumption: Memory-intensive engines like Presto drove up infrastructure costs.
They needed a single, unified solution that could deliver extreme performance, high compression, and robust analytics without the complexity and cost.
The Search for a Solution: Finding a Unified Powerhouse in Apache Doris
The team's search was guided by clear criteria: a high-performance, cost-effective, and secure solution that could handle massive-scale online analysis. Apache Doris stood out for several key reasons:
- Security and Control: As a state-level financial infrastructure, using a secure, controllable, and nationally compliant technology was a top priority.
- Powerful Analytics: Its MPP architecture and rich SQL support (including JOINs, subqueries, and views) were perfect for their complex security analysis needs, such as risk assessment and user behavior analysis.
- Extreme Performance: The promise of fast, MPP-based retrieval to solve their Elasticsearch query bottlenecks and high-throughput real-time writes was compelling.
- Low Cost: Doris offered a much higher data compression ratio and a simplified architecture, promising to drastically reduce both storage and operational costs.
The Results: A New Architecture with Dramatic Gains
The team consolidated their stack, replacing Elasticsearch, Hudi, Hive, and Presto with Apache Doris as the single, unified real-time analytics database. This simplified architecture immediately paid off, delivering transformative results across two key security scenarios: log search (querying raw logs across 100+ fields) and event dashboards (analyzing aggregated events with frequent updates).
The performance benchmarks, comparing Doris to their previous Elasticsearch setup across their top 9 query scenarios, were decisive:
- Up to 56x Faster Queries: Doris consistently outperformed Elasticsearch, with query response times improving by an average of 3x and up to a staggering 56x in the best case. Even complex full-text searches across hundreds of fields on petabyte-scale data became interactive.
- 50% Reduction in Storage Costs: Doris’s columnar storage engine and superior ZSTD compression (achieving 5-10x compression vs. ES's 1.5x) cut their storage footprint in half. They also plan to use Doris’s tiered storage capabilities to move cold data to object storage, further reducing costs by over 70%.
- 4x Increase in Write Throughput: By leveraging optimizations like single-replica import, single-tablet loading, and time-series compaction policies, Doris handled the high-volume data streams with significantly higher throughput and lower CPU consumption, even with extensive inverted indexing.
- Simplified Architecture & Developer Efficiency: Replacing a multi-system pipeline with a single Doris cluster eliminated data consistency issues. A complex user profiling task that previously took a 15-minute batch job could now be done with a single real-time SQL query in Doris, responding in sub-seconds.
- Enhanced Stability & High Availability: The OOM errors that plagued their Elasticsearch cluster during large-scale queries vanished. Doris handled petabyte-scale queries with stability, and its resource isolation features provided an extra layer of protection for system availability.
Conclusion: A Foundation for the Future
By migrating to Apache Doris, the payment platform not only solved its immediate performance and cost challenges but also built a modern, scalable, and efficient foundation for its future financial security needs.
Talk to Us
Want to see similar (or even greater) gains in your own data platform? Come join the Apache Doris community to connect with like-minded users and get expert support.
Looking for fully managed, cloud-native deployment options? Contact the VeloDB team!
