big data and technology stack for telecom company
TRANSCRIPT
Big Data Solution For
YTT Telecom
YTT Telecom
● Leading edge mobile voice, data and multimedia services
Company (63 M customers)
● Focus on R&D to enrich customer lives
● Adoption rate > 20%.
● 20% users switched to smart phone, > 3 times over 2013
● Need robust infrastructure to accept rapidly growing
network traffic
Telecom Eco-system
Data Data Everywhere
POS Data
Locations
Payments
Sensor Data
Customer Profiles
Weather
Shipments
Transactions
HR Records
Financial Records
Google+
Call Center Data
Click Stream
Text Messages
Online Forums
Video
Sharepoint
3rd Party Text Documents
Vel
oci
ty
Variety and Volume
YTT Data Challenge Matrix
Customer Touch Points
Portal
Store
FB
Yelp
Mailers Offers
Email SMS
Call Center
Network Data
• Billions of Call Detail Records
Location Data
• 60 TB of Location Data
Customer Data
• Millions of records for 63 M customers
Structured Unstructured
Problems Highlights
Smart Devices Data Needs Services Provided
• How to keep the customer happy/satisfied and reduce churn?
Customer Churn
• What strategies should YTT apply to store and analyze network data and resolve issues in real time?
Network Management
• How to run effective and targeted campaigns?
Marketing Campaign Efficiency
Key Trends
Expected Budget of Telecom Companies for Handling Big Data
Big Data Analytics for effective promotions
Big Data for Real Time Intelligence and control back into the network
Big Data Analytics to optimize network performance and reduce cost - T Mobile
Responses to Big Data Initiatives
Learn and Label : Segment Customers on the usage patterns, learn preferences, create labels and store with the profile. Create/offer suitable/customized plans.
Empower Customer Service : Allow a representative to help in near real time to resolve issues and make offers. A customer is rated/ranked on the basis of usage, payment history and interests.
Proactive Network Management : Detect network spikes, analyze dropped calls from CDR analysis, inform Customer Service in case of dropped calls to make a friendly call to the customer facing the problem.
Understanding Sentiment : Find out +ve/-ve sentiments on Social Networks/blogs etc. pre/post campaign to see the effectiveness.
Proposed Solution - Strategy
Gather internal/external
data
Ingest and standardize data
Apply S/W Tools to Prepare,
Process, Analyze and Export Data
Derive actionable insights
Determine actions on
results obtained
Solution Tech Stack
DATA SOURCES
CRM
Network data
SubscriberData
Billing Records
ERP
Product Related
Data
Customer Behavior
Click Stream Online chat Sensor DataSocial Media
System Operations
Server LogsCall Detail RecordsMerchant Listings
Signaling LogsProtocol Logs
Physical Layer
INGEST
SqoopFlume
HDFS.PutWeb.HDFS
Hadoop Distributed File System
Mapreduce Libraries Hbase Database
Ad-hoc Query Analysis
Oracle Workflow Scheduler
Pig Data AnalyticsHive Data
Warehouse
Multitenant Processing : YARN
Compute and HDFS Storage
Metadata Management : HCatalog
CDR AnalysisProactive network maintenanceBandwidth AllocationInfrastructure InvestmentOperational dashboardsCustomer scorecardsProduct DevelopmentAnalysis Layer
Proposed Solution Architecture
Call Detail Records
Network Logs
Tower CDRs
Call Center Record
QoS Reports, Billing InformationHadoop MapReduce + Pig Data Analytics
Hbase Database + Hive Queries
Social Media Data
Traffic Reports, Network Audit Reports
Hbase Database + Pig Data Analytics
Call Volume Reports, Routing Graphs
Traditional Datawarehouse + SQL
Customer Service Reports, Closed Loop reports
Hadoop MapReduce + COGSA
Sentiment Analysis Reports, Funnel Reports
Tower CDR LogCaller A;Caller B;Date;Time;Duration;Call Type;First Cell ID;Last Cell ID;Cell ID Zip9096714043;9163281129;8/4/2014;9:45:23;0;SMS-IN;405-799-20-36023;405-799-20-36361;947097276789858;9806154895;8/5/2014;9:50:11;1161;CALL-IN;405-799-20-36023;405-799-20-31611;94150…………………………………………………………………………………………………………………….....
MapReduce Job
Generates pairs of (tower id, # calls routed)
Tower Number #Calls (in000s) 405-805-105-60382 234405-805-127-10223 213405-805-127-33891 206405-805-127-10221 156405-805-105-60383 143…………………….. …..
Solution Design Mock-UpHandling Network Congestion
Deployment- Strategy
2 Week plan to validate proposed solutions for: Customer churn Network traffic Optimized Marketing spend
Identify Data Sources
Unify And Assemble Data
Clean and Enhance Data
Quality
Append Content
Build Analytics Analyze
Review Dashboard
OK to Proceed
Give us Access to YTT data and approval
Provide following resources: 1 Data Engineer 1 Network Engineer 1 Data Scientist 1 BI Engineer
Allow access to Cloud AWS infra (free trial) or equivalent
Proposed Solution Benefits
• Big data offers YTT Telecom a real opportunity to gain a more complete picture of their operations impacting their customers, and to further their innovation efforts.
• YTT’s focus on R&D is to enrich customer lives. This solution proposal is in consistence with their focus.
• Big data challenge can be met on the lines of the proposed Solution Architecture.
• YTT should incorporate new agile strategies into their organizational DNA fast so that it will gain a real competitive advantage over their slower rivals.
Summary
References:
1. TELECOMS.COM INTELLIGENCE INDUSTRY SURVEY 2014 – http://www.telecoms.com/wp-content/blogs.dir/1/files/2014/03/IndustrySurveyReport14_latest1.pdf