hadoop development in china mobile research institute
DESCRIPTION
Hadoop Development in China Mobile Research Institute, esp in HDFSTRANSCRIPT
![Page 1: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/1.jpg)
中国移动研究院的中国移动研究院的中国移动研究院的中国移动研究院的中国移动研究院的中国移动研究院的中国移动研究院的中国移动研究院的HadoopHadoopHadoopHadoopHadoopHadoopHadoopHadoop相关研发工作相关研发工作相关研发工作相关研发工作相关研发工作相关研发工作相关研发工作相关研发工作
中国移动研究院
王旭
![Page 2: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/2.jpg)
大云研发历程
中国移动启动“大云”“大云”研发计划,打造公司云计算核心竞争力“大云”计划是中国移动研究院为打造中国移动云计算基础设施而实施的关键技术研究及原型系统开发计划
目标为满足中国移动中国移动ITIT支撑系统支撑系统高性能、低成本、可扩展、高可靠性的IT计算和存储的需要为满足中国移动提供互联网业务和服务互联网业务和服务的需要
并行数据挖掘工具开发并行数据挖掘工具开发并行数据挖掘工具开发并行数据挖掘工具开发与应用试验与应用试验与应用试验与应用试验
系统改进系统改进系统改进系统改进、、、、完善与试验完善与试验完善与试验完善与试验
云计算技术路标云计算技术路标云计算技术路标云计算技术路标、、、、引入策略引入策略引入策略引入策略、、、、总体解决方案研究总体解决方案研究总体解决方案研究总体解决方案研究、、、、产品研产品研产品研产品研发发发发、、、、应用试验应用试验应用试验应用试验、、、、产业链培育产业链培育产业链培育产业链培育、、、、商务模式研究商务模式研究商务模式研究商务模式研究
研究研究研究研究Hadoop等等等等云计算关键技术云计算关键技术云计算关键技术云计算关键技术
开源架构开源架构开源架构开源架构/关键关键关键关键技术研究技术研究技术研究技术研究
系统评估系统评估系统评估系统评估
发布大发布大发布大发布大云云云云 0.5
搭建搭建搭建搭建1024
节点集群节点集群节点集群节点集群用闲置资源搭用闲置资源搭用闲置资源搭用闲置资源搭建 第 一 个建 第 一 个建 第 一 个建 第 一 个Hadoop平台平台平台平台
建设建设建设建设 256节点节点节点节点集群和分析工集群和分析工集群和分析工集群和分析工具具具具
2007.3 2008.3 2008.12 2009.8 2009.122007.7 2008.10
确定大云研确定大云研确定大云研确定大云研究方向究方向究方向究方向
2010.5
云计算大会上云计算大会上云计算大会上云计算大会上发布发布发布发布大云大云大云大云1.0
![Page 3: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/3.jpg)
建设1024节点规模的大规模实验室
实验室建设实验室建设实验室建设实验室建设• 2008年年年年9月完成了大规模运算实验室一期工程的月完成了大规模运算实验室一期工程的月完成了大规模运算实验室一期工程的月完成了大规模运算实验室一期工程的建设建设建设建设,,,,配置了配置了配置了配置了256台台台台PC服务器服务器服务器服务器,,,,初步建立了大初步建立了大初步建立了大初步建立了大规模运算平台研发和试验环境规模运算平台研发和试验环境规模运算平台研发和试验环境规模运算平台研发和试验环境
• 2009年年年年12月完成了大规模实验室二期扩容工程月完成了大规模实验室二期扩容工程月完成了大规模实验室二期扩容工程月完成了大规模实验室二期扩容工程建设建设建设建设
实验室部署实验室部署实验室部署实验室部署实验室环境实验室环境实验室环境实验室环境
• 节点节点节点节点::::1036个服务器个服务器个服务器个服务器;;;;5208个个个个CPU核核核核,,,,10T内内内内存存存存;;;;2.8P硬盘硬盘硬盘硬盘
• 交换机交换机交换机交换机::::9个万兆个万兆个万兆个万兆/千兆兼容以太网交换机千兆兼容以太网交换机千兆兼容以太网交换机千兆兼容以太网交换机,,,,树树树树形结构互联形结构互联形结构互联形结构互联
• 软件软件软件软件::::Centos Linux 5.4,,,,kernel 2.6.18, jdk
1.6, hadoop-0.20等等等等• 部署的应用部署的应用部署的应用部署的应用::::数据挖掘数据挖掘数据挖掘数据挖掘,,,,弹性计算平台弹性计算平台弹性计算平台弹性计算平台BC-EC,,,,结构化海量数据管理平台结构化海量数据管理平台结构化海量数据管理平台结构化海量数据管理平台HugeTable,,,,搜索引搜索引搜索引搜索引擎擎擎擎,,,,云存储云存储云存储云存储
![Page 4: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/4.jpg)
中国移动大云技术架构Application
• IT Supporting System of CMCC
• IDC and Internet Applications
Enabler
• BC-PDM: Cloud base Data Mining
• BC-NAS: File and Object Storage with
web Interface and REST API
• BC-SE: Search Engine
Platform
Syste
m M
anagem
ent :C
loudM
aste
r
Internet App IDC 。。。。。。。。。。。。
Clo
udSecurity
Sturcture Data Storage
HugeTable
Hadoop MapReduce
Data Mining
BC-PDM
Search Engine
BC-SE
CMCC IT Supporting Systems
Cloud Storage
BC-NAS
Platform
• MapReduce & HDFS: based on Hadoop
and with some extensions by CMRI
• HugeTable::::Structure Storage with
SQL interface
•oNest::::Object Storage for Web Apps.
• CloudMaster: System Management
Resource
• PC Server and SATA Disk based
• BC-EC: IaaS based on OpenNebula
• Based on FOSS: Linux, KVM, Xen
Syste
m M
anagem
ent :C
loudM
aste
r
Distributed Filesystem
based on
Hadoop HDFS
Elastic Computing: BC-EC
Linux, Xen/KVM
Clo
udSecurity
Hadoop MapReduce
with CMRI Extension
Object Storage
oNest
![Page 5: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/5.jpg)
大云与Hadoop
Parallel ETL and Data Mining
based on MapReduce
Search Engine based on
MapReduce
HugeTable (Structure data
storage for data warehouse)
based on Hive, HBase & MR
Syste
m M
anagem
ent :
Internet App IDC 。。。。。。。。。。。。
Clo
udSecurity
Sturcture Data Storage
HugeTable
Hadoop MapReduce
Data Mining
BC-PDM
Search Engine
BC-SE
CMCC IT Supporting Systems
Cloud Storage
BC-NAS
Development based on Hadoop
Development extending Hadoop
Syste
m M
anagem
ent :C
loudM
aste
r
Distributed Filesystem
based on
Hadoop HDFS
Elastic Computing: BC-EC
Linux, Xen/KVM
Clo
udSecurity
Hadoop MapReduce
with CMRI Extension
Object Storage
oNest
Development extending Hadoop
External facilities for Hadoop
Volume Management of
DataNode in HDFS
NameNode Cluster for HDFS
Multi-queue scheduler with
queue priority enhancement
Test tools for Hadoop HDFS
Inside job performance
evaluation tool
MapReduce Job Submission
Web Interface
![Page 6: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/6.jpg)
Development on Hadoop in CMRI
Contributing to Mainline
Online Volume Management of DataNode (by Wang Xu etc., HDFS-
1362)
Off-Tree and Opened
NameNode Cluster for HA (by Wang Xu, hosted in GitHub)
Off-Tree and not Maintained
hdfs-fuse (by Zhao Peng, hosted in Google Code)hdfs-fuse (by Zhao Peng, hosted in Google Code)
Multi-queue scheduler with queue priority enhancement (by Guo Leitao)
External Facilities
hadoop-test (by Wang Xu, hosted in Google Code)
MapReduce Job Submission Web Interface and Inside job performance
evaluation tool (by Guo Leitao, etc.)
Bug Fixes
![Page 7: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/7.jpg)
DataNode Online Volume Management
http://github.com/gnawux/hadoop-cmri
https://issues.apache.org/jira/browse/HDFS-1362
Current State:
Disk failure�Node
Decommission
Online VolumeOnline Volume
Management:
Online removal of
failed disk
Migrate Data in faild
volume if still
readable
Change Disk online
![Page 8: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/8.jpg)
NameNode Cluster
http://github.com/gnawux/hadoop-cmri [code]
http://gnawux.info/hadoop/2010/01/pratice-of-namenode-cluster-for-hdfs-ha/
http://gnawux.info/hadoop/2010/05/namenode-cluster-code-github/
![Page 9: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/9.jpg)
HDFS Stress Test
http://code.google.com/p/hadoop-test/
http://gnawux.info/hadoop/2010/01/a-simple-hdfs-performance-test-tool/
![Page 10: Hadoop development in China Mobile Research Institute](https://reader034.vdocuments.site/reader034/viewer/2022050815/54835507b47959ec0c8b4a59/html5/thumbnails/10.jpg)
谢谢关注谢谢关注谢谢关注谢谢关注谢谢关注谢谢关注谢谢关注谢谢关注
http://labs.chinamobile.com/cloud/