multi-tenant log analytics saas service using solr: presented by chirag gupta & srivatsan...
TRANSCRIPT
Multi-Tenant Log Analytics SaaS Service using SOLR Chirag Gupta (Principal Software Engineer)
Srivatsan Parthasarathy (Partner Software Engineer) Microsoft Corp.
Real-time search Patching
Configuration Change Tracking
Linux Solutions
Monitoring App Solutions
Community Gallery
Security Audit Data Protection Dashboards
Integrations/Collectors Log Analytics Service
Solution Platform
Microso' Log Analy0cs Solu0on Pla4orm
6
02
• 50K+ customers with 10M+ shards
Usage Statistics
• 20PB searchable documents • 188 million queries per week
7
02 Challenges • SOLR Cloud boDlenecks
• MulE-‐tenant support
• Performance issues with wide queries
IngesEon
Azure File Shares
SOLR Clusters
SOLR
Query
Cluster State
Microso' Log Analy0cs Cloud
Gateway
SOLR Query Merger
SOLR Cluster
Query Inges0on
Predictable Load Single Customer/
Workload
SOLR Workloads
Light Load Moderate Load Heavy Load
Long tail of large and small customers Growing/shrinking without warning
IngesEon
SOLR Clusters
Azure File Shares
Cold Cluster1 SOLR
Query
Microso' Log Analy0cs Workload Management
Cluster State
Gateway
SOLR Query Merger
Management
Query
Cold Cluster1 SOLR
Hot SOLR Clusters
Azure File Shares
Cold Cluster1 SOLR
Cold Cluster
Microso' Log Analy0cs Cold Query
Cluster State
Gateway
SOLR Query Merger
14
02 Recap • Used external durable storage vs replicaEon
• Central scoped config with ApplicaEon Load Balancing vs
Zookeeper
• Shard Lifecycle Management as a separate component
• Cold Storage based Queries
15
02 Next Steps • Further opEmize SOLR for Log AnalyEcs Scenario
• IngesEon pipeline
• AnalyEcs queries