Elastic Scaling for Efficient Resource Management

Elastic architecture

As a data analytics infrastructure, elastic resource management has always been its eternal pursuit. From all-in-one appliances to distributed clusters, resources can be linearly scaled. From distributed clusters to today's cloud-native elastic services, fast elastic expansion and contraction can be completed at the minute level according to load requirements. The ultimate elastic architecture makes resource management more flexible and efficient, reducing resource usage costs.

The ultimate elastic architecture relies on three separations. The separation of compute and compute enables more fine-grained management of computing resources; the tiered storage of data with different hotness achieves a significant reduction in storage costs without losing storage performance; the separation of storage and compute allows computing and storage achieves true independent expansion and contraction.

Key capabilities

Compute-Compute Separation

Three workload isolation mechanisms

  • Resource Group: hard isolation that divides backend nodes into resource groups
  • Workload Group: soft isolation of computing workloads in backend processes
  • Compute Node: do not store data but query data from external data lakes or databases
Storage-Storage Separation

Tiered storage for lower storage cost of cold data

  • Move data from SSD to HDD, object storage / HDFS
  • Support both relative and absolute cooldown time
  • Support cold data caching
Storage-Compute Separation

Cloud-native services that separate storage and compute

  • Object storage as shared storage and high-speed local caching
  • Multiple clusters to isolate computing workloads
  • Elastic scaling of compute nodes