Multi-Cluster Elasticsearch Management in Distributed Search Applications

As big data applications continue to evolve and cloud-native architectures gain popularity, many large-scale search and analytics platforms now rely on aspects of Elasticsearch. But, as other multiple cluster and distributed systems have experienced challenges with consistency, latency, fault tolerance, and resource management, so too has Elasticsearch. This review explored various multi-cluster Elasticsearch management techniques, such as Cross-Cluster Search (CCS) and Cross-Cluster Replication (CCR). We put forth a unique theoretical model called Adaptive Federated Cluster Orchestration (AFCO) that incorporates: 1. AI-powered orchestration; 2. federated policy enforcement mechanisms; and 3. policy monitoring capabilities. Our review also provided future directions for privacy-preserving search, autonomous orchestration and federated learning to help increase the adaptability and resiliency of distributed Elasticsearch systems.