As businesses expand globally and digital operations deepen, the scale and complexity of log systems have grown significantly. For Advance Intelligence Group, a Singapore-based AI and fintech company, its original OpenSearch-based log platform began struggling to meet requirements around cost control, performance, and maintainability.
After evaluating alternatives, including ClickHouse, Loki, and others, the Group selected Apache Doris as the new core storage solution for its log management system. This decision was influenced by Apache Doris's high performance in both search and analytics, and provides a good case study for users looking for an alternative to OpenSearch and Elasticsearch for log analytics.
After switching from OpenSearch to Apache Doris, the Group has seen remarkable results: More than 50% in comprehensive cost reduction, a 5x improvement in query performance, near-real-time log ingestion, and flexible operational and maintenance.
Background on Advance Intelligence Group
Advance Intelligence Group is a Singapore-headquartered AI and fintech company. Its enterprise arm, ADVANCE.AI, offers AI-powered digital identity verification, fraud detection, and risk management solutions for banks, fintechs, and e-commerce platforms. The Group also operates Atome, a buy-now-pay-later consumer service, and Ginee, an omnichannel e-commerce SaaS platform for merchants. Currently, the Group serves over 300 enterprise clients, 240,000 merchants, and 58 million users across regions.
Original Architecture with OpenSearch and Pain Points
As the Advanced Intelligence Group rapidly scaled its business, system logs also grew exponentially. The original log analysis platform based on OpenSearch began hitting structural bottlenecks and could no longer keep pace with the increasing volume and complexity of log processing needs. The Group has identified several key pain points:
- Rapid Cost Surge: As the log volume continues to grow, the computing and storage pressure have increased significantly. Server resource consumption, object storage costs, and node local storage requirements contributed to the cost increase
- Slow Query Performance: In high-concurrency and complex multi-dimensional analytics scenarios, OpenSearch query response speed degraded significantly, affecting real-time performance and the company's operational efficiency.
- High Operational Complexity: Each scaling operation not only increases resource consumption but also further increases maintenance complexity, raising real-time and operational efficiency risks.
To address the pain points above, the project was launched to ensure high availability, real-time performance, and query performance of the log system while reducing the total cost of ownership.
Why Migrate from OpenSearch to Apache Doris
During the technical evaluation phase, Advance Intelligence Group tested various log system solutions, including ClickHouse, Apache Doris, Loki, etc. After technical evaluation and testing, the company chose Apache Doris as the core component of the new-generation log analysis platform, primarily based on Apache Doris's key advantages, including efficient columnar compression, near-real-time ingestion, flexible scaling mechanism, powerful search capabilities, good compatibility, and low operational and maintenance costs.
The following are the conclusions of a comprehensive technical comparison between OpenSearch and Apache Doris:
| Comparison Dimension | OpenSearch (Original Architecture) | Apache Doris (New Architecture) |
|---|---|---|
| Architecture Type | Distributed search engine (derived from Elasticsearch) | MPP (Massively Parallel Processing) columnar database |
| Production Node Deployment Structure | Coordinating ×2 (request aggregation)Master ×2 (metadata maintenance, master election, and cluster management)Dashboards ×1 (providing data visualization functions)Data ×26 (indexing, querying, storage) | FE ×3 (high availability, SQL parsing and scheduling, metadata management)BE ×12 (storage and query execution) |
| Query Engine Type | Inverted index (suitable for full-text search) | Vectorized execution engine (suitable for structured aggregation) and supports inverted index (suitable for full-text search) |
| Query Performance | Queries involve multiple indexes, slow response under high concurrency, and poor multi-dimensional aggregation efficiency | The query optimizer supports predicate pushdown, and the performance in aggregation scenarios is far superior to OpenSearch |
| Real-time | Minute-level, affected by refresh cycles and load | Near-real-time, logs can be queried almost immediately after writing, suitable for near-real-time log analysis needs |
| Data Compression Efficiency | Medium, default LZ4 compression | High, LSM-Tree architecture + ZSTD compression |
| Scalability | Scaling involves shard redistribution and replica synchronization, which is slow and likely to cause cluster instability | One-click flexible scaling, automatic replica balance, no need to rebuild indexes |
| Storage and Lifecycle Management | High storage cost.Log retention relies on external cold storage, increasing additional maintenance costs | High compression ratio reduces storage cost.Mature backup and recovery mechanisms, and supports hot-cold tiering and object storage with flexible lifecycle management |
| Operational Complexity | Complex configuration, high cost for node expansion and tuning | Easy-to-use management tools. |
| Visualization Support | Kibana / OpenSearch Dashboards | DorisManager Studio Search and Analytics / Third-party BI / Self-developed visualization tool. |
| Cost Control Capability | Large resource requirements and frequent expansion lead to high storage and computing costs | Reduces the number of nodes and object storage space, with significant cost savings |
| Easy to Use | Complex DSL and hard performance tuning | Compatible with MySQL protocol, easy to development/operation and maintenance |
Cost Optimization: How Apache Doris Helped to Cut Log System Costs by 50%
In the Advance Intelligence Group use case, Apache Doris helped cut down 50% of the cost of OpenSearch without sacrificing performance or features through the following measures:
- CPU and Disk Storage Reduction: Apache Doris leverages columnar storage, optimized index structure, and advanced compression, drastically reducing hardware requirements, providing up to 80% savings from reduced CPU cycles and disk volume.
- Inverted Index Support: Apache Doris can also maintain search SLAs, as it supports inverted indexes and full-text search like OpenSearch or Elasticsearch, and it runs search queries faster, thanks to the index optimizations for logs.
- Faster search and aggregation queries: Apache Doris is designed for real-time analytics, supporting a wide range of aggregations, often used in observability. And for search queries, Doris implements an inverted index in a way that's optimized for log search and optimized for the
TopNsearch queries likeSELECT * FROM t WHERE message MATCH 'error' ORDER BY time DESC LIMIT 100. The result is that Doris is 2x faster for search queries and 10x faster for aggregation queries.
Migration Steps and Results
The migration of the log system from OpenSearch to Apache Doris involves not only data and service migration but also adjustments to query APIs and operational and maintenance processes. We divided the entire process into the following steps:
- Data Model Design and Mapping: Design the corresponding Apache Doris table structure, and conduct standardized modeling according to log types (e.g., msg, logger, level, etc.).
- Log Collection Channel Transformation: Replace the original pipeline of Kafka → Logstash → OpenSearch to Kafka → Logstash → Doris Stream Load to achieve high-throughput and low-latency data ingestion.
- Query Rewrite: Convert OpenSearch DSL queries into Doris SQL and optimize them using partition pruning, column pruning, and predicate pushdown.
- Performance and Stress Testing: Compare ingestion and query performance under pressure in the test environment. After verifying stability, migrate each log module gradually until the full replacement is achieved.
After Advance Intelligence Group migrated from OpenSearch to Apache Doris, it has seen significant optimizations in the number of servers, storage, and query performance:
| OpenSearch (Before Migration) | Apache Doris (After Migration) | |
|---|---|---|
| Number of Servers | 31 | 15 |
| Cost Savings | N/A | 50% |
| Query Response Time | Average 5 seconds | Average 1 second |
| Ingestion Latency | Seconds to minutes | Sub second |
In addition, Apache Doris supports flexible elastic scaling with an efficient data compression mechanism. Even if log data volume continues to grow, it can significantly reduce storage, operations, and maintenance costs.
Summary
The migration of the log system from OpenSearch to Apache Doris has not only achieved the goal of significantly reducing cost, but also laid a solid foundation for future system scaling, query efficiency, and maintainability.
The core benefits include:
- Comprehensive cost reduction of over 50% between compute and storage
- Improving the query performance to 5x of the original system
- Near real-time log writing, supporting more time-critical business functions
- A simpler system architecture, stronger maintainability, and support for more flexible operation and maintenance strategies
Through this practice, Advance Intelligence Group has verified the benefits of combining search + MPP analytics engine with columnar storage. In use cases that require extreme performance and cost-effectiveness, Apache Doris provides a feasible and efficient alternative to OpenSearch or Elasticsearch, offering new solutions for log and observability platforms.
Join the Apache Doris community on Slack and connect with Doris experts and users. If you're looking for a fully managed Apache Doris cloud service, contact our team. For more user stories and technical deep dives on observability, visit our blog section here.
