2012 09-08-josug-jeff
TRANSCRIPT
1
2012.09.08OpenStack Japan
Zheng Xu
openstackOpen source software to build public and private clouds.
Hadoop on OpenStack Swift
OSC 2012 Tokyo
- Experiment of using swift as storage for Apache Hadoop
2
Self introduction
● Software designer(engineer) for embedded system and web system(60%hobbit, 40%job).
● Major: openstack, linux, web browser, html, epub, OSS
● Contact● @xz911● https://www.facebook.com/xuzheng2001
3
Abstract
● This slide is to introduce how to use OpenStack Swift as storage service for Apache Hadoop instead of HDFS(which is storage service of Hadoop project).
● This slide is based on http://bigdatacraft.com/archives/349, and really appreciate Constantine Peresypkin and David Gruzman for providing their idea and implementation.
4
Agenda
● OpenStack Swift
● Apache Hadoop and HDFS
● Experiment of replacing HDFS by OpenStack Swift
5
What is OpenStack and Swift
From http://www.openstack.org/
6
What is OpenStack and Swift
httpUser Application
Proxy Server Proxy Server
Container Server Container Server Container Server
Object Server Object Server Object Server Object Server
Account Server Account Server Account Server
http
http
7
What is OpenStack and Swift
● OpenSource written in Python● diversity
● Swift can be a part of OpenStack or an individual service it self.
● zones, devices, partitions, and replicas● No SPOF
8
Agenda
● OpenStack Swift
● Apache Hadoop and HDFS
● Experiment of replacing HDFS by OpenStack Swift
9
Apache Hadoop and HDFS
From http://hadoop.apache.org/
10
Apache Hadoop and HDFS
User Application
Name Node
Data Node Data Node Data Node
Map-Reduce
Hive
11
Agenda
● OpenStack Swift
● Apache Hadoop and HDFS
● Experiment of replacing HDFS by OpenStack Swift
12
Experiment (Concept)
Name Node
Data Node Data Node Data Node
Map-Reduce
Hive
User Application
13
Experiment (Concept)
Data Node Data Node
Map-Reduce
Swift
User Application
Hivejava-cloudfiles java-cloudfiles
java-cloudfiles
http
14
Experiment (Software)● Swift v1.6
● https://github.com/openstack/swift.git● r21616cf, Jul 25
● Java Client java-cloudfiles● https://github.com/rackspace/java-cloudfiles● r0807fa6, Jun 4
● Apache Hadoop● 1.0.3
● Swift fs for Apache Hadoop(just part of following source code)● https://github.com/Dazo-org/hadoop-common.git (branch-0.20-security-
205.swift )
15
Experiment (infra)
192.168.0.9192.168.0.4
16
Experiment(install swift)
● Install swift based on http://docs.openstack.org/developer/swift/development_saio.html
● Do not forget to set bind_ip of proxy-server.conf● 192.168.0.9 in my case
● Suppose we have username as "test:tester" with password as "testing", the account name is AUTH_test and have some container based on steps in above Url.
17
Experiment (cloudfiles)
● Run "ant compile"● Change cloudfiles.properties to following
# Auth info
auth_url=http://192.168.0.9:8080/auth/v1.0
auth_token_name=X-Auth-Token
#auth_user_header=X-Storage-User
#auth_pass_header=X-Storage-Pass
# user properties
username=test:tester
password=testing
# cloudfs properties
version=v1
connection_timeout=15000
18
Experiment(cloudfiles)
● Connect cloudfiles to swift(this is option)● Change cloudfiles.sh as following and run it to try
connection with swift#!/bin/sh
export CLASSPATH=lib/httpcore-4.1.4.jar:lib/commons-cli-1.1.jar:lib/httpclient-4.1.3.jar:lib/commons-lang-2.4.jar:lib/junit.jar:lib/commons-codec-1.3.jar:lib/commons-io-1.4.jar:lib/commons-logging-1.1.1.jar:lib/log4j-1.2.15.jar:dist/java-cloudfiles.jar:.
java com.rackspacecloud.client.cloudfiles.sample.FilesCli $@
19
Experiment (cloudfiles)
● Packaging java-cloudfiles to jar file for Apache Hadoop (clone java-cloudfiles to ~/java-cloudfiles)● We need to put *.properties into java-cloudfiles.jar
$ ant package
$ cd cloudfiles/dist
$ cp ../*.properties .
$ rm java-cloudfiles.jar
$ jar cvf java-cloudfiles.jar ./*
20
Experiment (hadoop)● Prepare
● download hadoop to ~/hadoop-1.0.3 (newest stable version of original hadoop) and git clone https://github.com/Dazo-org/hadoop-common.git to ~/hadoop-common (old hadoop source code with swift fs plugin)
● At ~/hadoop-1.0.3 (copy java-cloudfiles and related library to hadoop lib folder)– cd lib;cp ~/java-cloudfiles/cloudfiles/dist/java-cloudfiles.jar .– cp ~/java-cloudfiles/lib/httpc* .
21
Experiment (setting hadoop)
● ./hadoop-1.0.3/src/core/core-default.xml● Add following to make hadoop can recognize
handle "swift://" schema to SwiftFileSystem class<property>
<name>fs.swift.impl</name> <value>org.apache.hadoop.fs.swift.SwiftFileSystem</value>
<description>The FileSystem for swift: uris.</description>
</property>
22
Experiment (hadoop)
● Copy implementation for swift fs to hadoop 1.0.3 and build● cp -R ../hadoop-
common/src/core/org/apache/hadoop/fs/swift ./src/core/org/apache/hadoop/fs
● ant
23
Experiment(hadoop setting)
● ./conf/core-site.xml (part1)● Add following property for example
<property>
<name>fs.swift.userName</name>
<value>test:tester</value>
</property>
24
Experiment (hadoop setting)● ./conf/core-site.xml (part2)
● Add following property for example
<property>
<name>fs.swift.userPassword</name>
<value>testing</value>
</property>
<property>
<name>fs.swift.acccountname</name>
<value>AUTH_test</value>
</property>
25
Experiment (hadoop setting)● ./conf/core-site.xml (part3)
● Add following property for example
<property>
<name>fs.swift.authUrl</name>
<value>http://192.168.0.9:8080/auth/v1.0</value>
</property>
<property>
<name>fs.default.name</name>
<value>swift://192.168.0.9:8080/v1/AUTH_test</value>
</property>
26
Experiment (check swift fs)
● At this time, we should can list account information via following command● ./bin/hadoop -fs -ls /● or ./bin/hadoop fs -put ./conf/core-site.xml
/test_container/core-site.xml (test_container is a test container created after swift installed)
27
Finally
● We installed swift for storage service of hadoop● We built origin java-cloudfiles and created
packages for hadoop● We copied fs.swift plugin from
https://github.com/Dazo-org/hadoop-common.git to new hadoop source tree and build hadoop
● We set up core-site.xml of hadoop to connect to swift via java-cloudfiles
28
Thank you for listening.