datanode 硬碟空間配置 軟體貨櫃主機. datanode 硬碟空間配置 ( 一 ) $ ssh dna1 $ df...
DESCRIPTION
DataNode 硬碟空間配置 ( 二 ) $ sudo nano /opt/conf/A/hdfs-site.xml :: dfs.datanode.du.reserved $ stophdfs a $ starthdfs a $ ssh nna hdfs dfsadmin -report :: Name: :50010 (dna1) Hostname: dna1 Rack: /17/10 Decommission Status : Normal Configured Capacity: (10.32 GB) DFS Used: (32 KB) Non DFS Used: (6.50 GB) DFS Remaining: (3.83 GB) DFS Used%: 0.00% DFS Remaining%: 37.08%TRANSCRIPT
DataNode 硬碟空間配置 軟體貨櫃主機
DataNode 硬碟空間配置 ( 一 )
$ ssh dna1
$ df -hFilesystem Size Used Avail Use% Mounted onrootfs 19G 5.6G 12G 32% /none 19G 5.6G 12G 32% / :: $ hdfs dfsadmin -report ::Name: 172.17.10.20:50010 (dna1)Hostname: dna1Rack: /17/10Decommission Status : NormalConfigured Capacity: 19674116096 (18.32 GB)DFS Used: 61440 (60 KB)Non DFS Used: 6972833792 (6.49 GB)DFS Remaining: 12701220864 (11.83 GB)DFS Used%: 0.00%DFS Remaining%: 64.56% ::
DataNode 硬碟空間配置 ( 二 )
$ sudo nano /opt/conf/A/hdfs-site.xml :: <property> <name>dfs.datanode.du.reserved</name> <value>8589934592</value> <!-- 8 G --> </property>
$ stophdfs a$ starthdfs a
$ ssh nna hdfs dfsadmin -report ::Name: 172.17.10.20:50010 (dna1)Hostname: dna1Rack: /17/10Decommission Status : NormalConfigured Capacity: 11084181504 (10.32 GB)DFS Used: 32768 (32 KB)Non DFS Used: 6974406656 (6.50 GB)DFS Remaining: 4109742080 (3.83 GB)DFS Used%: 0.00%DFS Remaining%: 37.08%
管理 YARN 運算資源軟體貨櫃主機
管理 YARN 運算資源$ sudo nano /opt/conf/A/yarn-site.xml
<property> <name>yarn.nodemanager.resource.memory-mb</name> <value>1024</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>1</value> </property>
$ curl http://rma:8088/ws/v1/cluster/metrics{"clusterMetrics":{"appsSubmitted":0,"appsCompleted":0,"appsPending":0,"appsRunning":0,"appsFailed":0,"appsKilled":0,"reservedMB":0,"availableMB":2048,"allocatedMB":0,"reservedVirtualCores":0,"availableVirtualCores":2,"allocatedVirtualCores":0,"containersAllocated":0,"containersReserved":0,"containersPending":0,"totalMB":2048,"totalVirtualCores":2,"totalNodes":2,"lostNodes":0,"unhealthyNodes":0,"decommissionedNodes":0,"rebootedNodes":0,"activeNodes":2}}
管理 YARN 運算資源設定 MapReduce 程式記憶體需求$ cat /opt/conf/A/mapred-site.xml
<property> <name>yarn.app.mapreduce.am.resource.mb</name> <value>512</value> </property>
軟體貨櫃主機新增 Node Manager
新增 Node Manger
$ sudo nano /opt/hosts-0.2 ::172.17.10.52 nma3 # node manager ::$ dkcreate a ::Warning: Permanently added 'nma3,172.17.10.52' (ECDSA) to the list of known hosts.nma3 created ::
$ dkstart a.yarnbigred@dk:~$ dkstart a.yarnRma RunningNma1 RunningNma2 RunningNma3 starting java version “1.7.0_79” Scala compiler version 2.11.5 – Copyright 2002-2013, LAMP/EPFL
$ startyarn a
新增 Node Manger
$ ssh rma yarn node -list -all15/08/19 00:29:26 INFO client.RMProxy: Connecting to ResourceManager at rma/172.17.10.30:8032Total Nodes:3 Node-Id Node-State Node-Http-Address Number-of-Running-Containers nma1:55970 RUNNING nma1:8042 0 nma2:44551 RUNNING nma2:8042 0 nma3:35229 RUNNING nma3:8042 0
$ curl http://rma:8088/ws/v1/cluster/metrics{"clusterMetrics":{"appsSubmitted":0,"appsCompleted":0,"appsPending":0,"appsRunning":0,"appsFailed":0,"appsKilled":0,"reservedMB":0,"availableMB":3072,"allocatedMB":0,"reservedVirtualCores":0,"availableVirtualCores":3,"allocatedVirtualCores":0,"containersAllocated":0,"containersReserved":0,"containersPending":0,"totalMB":3072,"totalVirtualCores":3,"totalNodes":3,"lostNodes":0,"unhealthyNodes":0,"decommissionedNodes":0,"rebootedNodes":0,"activeNodes":3}}
設定 YARN 分散運算Node Manager 白名單
軟體貨櫃主機
設定 YARN 分散運算 – Node Manager 白名單$ sudo nano /opt/conf/A/yarn.allownma1nma2
$ sudo nano /opt/conf/A/yarn-site.xml :: <property> <name>yarn.resourcemanager.nodes.include-path</name> <value>/opt/conf/A/yarn.allow</value> </property>
啟用 Node Manager 白名單$ stopyarn a
$ startyarn a
$ ssh rma yarn node -list -all15/08/19 00:45:53 INFO client.RMProxy: Connecting to ResourceManager at rma/172.17.10.30:8032Total Nodes:2 Node-Id Node-State Node-Http-Address Number-of-Running-Containers nma2:34726 RUNNING nma2:8042 0 nma1:48948 RUNNING nma1:8042 0
修改 Node Manager 白名單$ sudo nano /opt/conf/A/yarn.allownma1nma2Nma3
$ ssh nma3 yarn-daemon.sh start nodemanager
$ yarn rmadmin -refreshNodes
$ yarn node -list -all15/08/19 00:52:35 INFO client.RMProxy: Connecting to ResourceManager at rma/172.17.10.30:803215/08/19 00:52:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableTotal Nodes:3 Node-Id Node-State Node-Http-Address Number-of-Running-Containers nma3:37007 RUNNING nma3:8042 0 nma2:34726 RUNNING nma2:8042 0 nma1:48948 RUNNING nma1:8042 0
Pig 分析工具軟體貨櫃主機
下載資料$ ssh ds01@cla01
$ wget http://tobala.net/x/download/hypermarket.csv--2015-08-19 02:44:02-- http://tobala.net/x/download/hypermarket.csvResolving tobala.net (tobala.net)... 69.89.27.215Connecting to tobala.net (tobala.net)|69.89.27.215|:80... connected.HTTP request sent, awaiting response... 200 OK ::2015-08-19 02:44:11 (67.6 KB/s) - ‘hypermarket.csv’ saved [15944/15944]
$ hdfs dfs -put hypermarket.csv hypermarket.csv
啟動 pig
$ piggrunt> clear;grunt> a = load '/data/hypermarket.csv' using PigStorage(',');2015-08-19 03:03:35,233 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFSgrunt> dump a;
grunt> b = foreach a generate $0,$1,$2,$3,$4,$5;grunt> dump b;
grunt> c = order b by $0 ASC;grunt> dump c;
grunt> store d into 'customer.csv' using PigStorage(',');
執行 customer.pig
$ nano customer.piga = load 'hypermarket.csv' using PigStorage(',');b = foreach a generate $0,$1,$2,$3,$4,$5;c = order b by $0 ASC;d = filter c by $0 != ' 會員編號 ';store d into 'customer.csv' using PigStorage(',');$ pig -f customer.pig
$ pig -f customer.pig
HDFS Balancer
軟體貨櫃主機
參考文章 1. Hadoop HDFS Balancer Explainedhttp://www.swiss-scalability.com/2013/08/hadoop-hdfs-balancer-explained.html
上載檔案至 HDFS $ ssh cla01
$ dd if=/dev/zero of=foo.bar bs=1M count=2501500+0 records in1500+0 records out1572864000 bytes (1.6 GB) copied, 178.525 s, 8.8 MB/s
$ hdfs dfs -put foo.bar /tmp
設定 HDFS Balancer 傳輸量$ sudo nano /opt/A/hdfs-site.xml :: <property> <name>dfs.datanode.balance.bandwidthPerSec</name> <value>52428800</value> <!-- 50 M --> </property>
啟動 HDFS Balancer ( 一 )
$ hdfs dfsadmin -reportConfigured Capacity: 5554536448 (5.17 GB) ::Name: 172.18.1.20:50010 (dna1) ::Configured Capacity: 566042624 (539.82 MB)DFS Used: 528461824 (503.98 MB)Non DFS Used: 0 (0 B)DFS Remaining: 37580800 (35.84 MB)DFS Used%: 93.36%DFS Remaining%: 6.64% ::Name: 172.17.10.30:50010 (dna2) ::Configured Capacity: 2494246912 (2.32 GB)DFS Used: 281980935 (268.92 MB)Non DFS Used: 0 (0 B)DFS Remaining: 2212265977 (2.06 GB)DFS Used%: 11.31%DFS Remaining%: 88.69% ::* 上面 dna1 的 DFS Used% 值為 93.36%, 代表可儲存空間已快用完
啟動 HDFS Balancer ( 二 )
$ hdfs balancer -threshold 30 ("-threshold" : Percentage of disk capacity) ::15/07/10 20:17:38 INFO balancer.Balancer: 1 over-utilized: [172.18.1.20:50010:DISK]15/07/10 20:17:38 INFO balancer.Balancer: 0 underutilized: []15/07/10 20:17:38 INFO balancer.Balancer: Need to move 237.59 MB to make the cluster balanced.15/07/10 20:17:38 INFO balancer.Balancer: Decided to move 161.95 MB bytes from 172.18.1.20:50010:DISK to 172.17.10.30:50010:DISK15/07/10 20:17:38 INFO balancer.Balancer: Will move 161.95 MB in this iteration2015/7/10 下午 08:17:39 0 0 B 237.59 MB 161.95 MB ::15/07/10 20:17:48 INFO balancer.Balancer: 1 over-utilized: [172.18.1.20:50010:DISK]15/07/10 20:17:48 INFO balancer.Balancer: 0 underutilized: []15/07/10 20:17:48 INFO balancer.Balancer: Need to move 237.59 MB to make the cluster balanced.15/07/10 20:17:48 INFO balancer.Balancer: Decided to move 161.95 MB bytes from 172.18.1.20:50010:DISK to 172.17.10.30:50010:DISK ::* 172.18.1.20 是 dna1 的 IP 位址
HDFS Federation
軟體貨櫃主機
參考文章 1. HDFS Federation Configurationhttp://datadotz.com/hdfs-federation/
建立第二個 HDFS 分散檔案系統建立第二個 HDFS 所有貨櫃主機$ sudo nano /opt/hosts-0.2
$ dkcreate b ::clb1 created : dsb01 dsb02nnb created : dsb01 dsb02dnb1 createddnb2 createdrmb creatednmb1 creatednmb2 created
啟動第二個 HDFS 所有貨櫃主機$ dkstart b.hdfsnnb starting java version "1.7.0_79" Scala compiler version 2.11.5 -- Copyright 2002-2013, LAMP/EPFL
dnb1 starting java version "1.7.0_79" Scala compiler version 2.11.5 -- Copyright 2002-2013, LAMP/EPFL
dnb2 starting java version "1.7.0_79" Scala compiler version 2.11.5 -- Copyright 2002-2013, LAMP/EPFL
$ formathdfs b myringformat (yes/no) yesnnb format oknnb clean sndnb1 clean dndnb2 clean dn
設定第二個 HDFS 分散檔案系統$ sudo nano /opt/conf/B/hdfs-site.xml ::
<property> <name>fs.defaultFS</name> <value>hdfs://nnb:8020</value> </property> <property> <name>fs.default.name</name> <value>hdfs://nnb:8020</value> </property>
設定第二個 HDFS 分散檔案系統$ sudo nano /opt/conf/B/hdfs-site.xml :: <property> <name>dfs.nameservices</name> <value>hdfs1,hdfs2</value> </property> <property> <name>dfs.namenode.rpc-address.hdfs1</name> <value>nna:8020</value> </property> <property> <name>dfs.namenode.rpc-address.hdfs2</name> <value>nnb:8020</value> </property>
啟動 HDFS 分散檔案系統$ starthdfs bstarting namenode, logging to /tmp/hadoop-bigred-namenode-nnb.outstarting secondarynamenode, logging to /tmp/hadoop-bigred-secondarynamenode-nnb.outstarting datanode, logging to /tmp/hadoop-bigred-datanode-dnb1.outstarting datanode, logging to /tmp/hadoop-bigred-datanode-dnb2.out
$ ssh nnb $ hdfs dfsadmin -report ::-------------------------------------------------Live datanodes (2):
Name: 172.17.20.30:50010 (dnb1)Hostname: dnb1 ::Name: 172.17.20.31:50010 (dnb2)Hostname: dnb2 ::
HDFS Federation
軟體貨櫃主機
設定第一個 HDFS 分散檔案系統$ sudo nano /opt/conf/A/hdfs-site.xml :: <property> <name>dfs.nameservices</name> <value>hdfs1,hdfs2</value> </property> <property> <name>dfs.namenode.rpc-address.hdfs1</name> <value>nna:8020</value> </property> <property> <name>dfs.namenode.rpc-address.hdfs2</name> <value>nnb:8020</value> </property>
重新格式化第一個 HDFS
關閉 NameNode$ hadoop-daemon.sh stop namenodestopping namenode
$ hdfs namenode -format -clusterID myring ::15/07/10 16:40:34 INFO namenode.NNConf: XAttrs enabled? true15/07/10 16:40:34 INFO namenode.NNConf: Maximum size of an xattr: 16384Re-format filesystem in Storage Directory /home/pi/nn ? (Y or N) y ::15/07/10 16:41:34 INFO util.ExitUtil: Exiting with status 015/07/10 16:41:34 INFO namenode.NameNode: SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NameNode at nna/172.18.1.10************************************************************/
清除 DataNode 的資料目錄$ ssh dna1pi@dna1's password:
$ hadoop-daemon.sh stop datanodestopping datanode$ rm -r dn$ exit
$ ssh dna2pi@dna1's password:
$ hadoop-daemon.sh stop datanodestopping datanode$ rm -r dn$ exit
$ ssh dna3pi@dna1's password:
$ hadoop-daemon.sh stop datanodestopping datanode$ rm -r dn$ exit
啟動第一個 HDFS 分散檔案系統$ hadoop-daemon.sh start namenodestarting namenode, logging to /tmp/hadoop-pi-namenode-nna.out
$ ssh dna1pi@dna1's password:$ hadoop-daemon.sh start datanodestarting datanode, logging to /tmp/hadoop-pi-datanode-dna1.out$ exit
$ ssh dna2pi@dna2's password:$ hadoop-daemon.sh start datanodestarting datanode, logging to /tmp/hadoop-pi-datanode-dna2.out$ exit
$ ssh dna3pi@dna3's password:$ hadoop-daemon.sh start datanodestarting datanode, logging to /tmp/hadoop-pi-datanode-dna3.out$ exit
HDFS Federation
軟體貨櫃主機
參考文章 1. HDFS Federation Configurationhttp://datadotz.com/hdfs-federation/
設定第一個 HDFS 的 Hadoop Client
$ sudo nano /opt/conf/A/core-site.xml :: <property> <name>fs.default.name</name> <value>viewfs://myring/</value> </property> <property> <name>fs.viewfs.mounttable.myring.link./hdfs1</name> <value>hdfs://nna:8020</value> </property> <property> <name>fs.viewfs.mounttable.myring.link./hdfs2</name> <value>hdfs://nnb:8020</value> </property>
使用 HDFS 聯邦分散檔案系統$ ssh cl01$ hdfs dfs -ls /Found 2 items-r-xr-xr-x - bigred bigred 0 2015-07-11 01:38 /hdfs1-r-xr-xr-x - bigred bigred 0 2015-07-11 01:38 /hdfs2
$ hdfs dfs -ls -d /hdfs1drwxr-xr-x - bigred supergroup 0 1970-01-01 08:00 /hdfs1
$ hdfs dfs -mkdir /hdfs1/abc
$ hdfs dfs -mkdir /hdfs2/xyzmkdir: Permission denied: user=bigred, access=WRITE, inode="/":pi:supergroup:drwxr-xr-x
設定第二個 HDFS 的 Hadoop Client
$ sudo nano /opt/pi/hadoop-2.6.0/etc/hadoop/core-site.xml :: <property> <name>fs.default.name</name> <value>viewfs://myring/</value> </property> <property> <name>fs.viewfs.mounttable.myring.link./hdfs1</name> <value>hdfs://nna:8020</value> </property> <property> <name>fs.viewfs.mounttable.myring.link./hdfs2</name> <value>hdfs://nnb:8020</value> </property>
</configuration>
使用 HDFS 聯邦分散檔案系統$ ssh cla01$ hdfs dfs -ls /Found 2 items-r-xr-xr-x - bigred bigred 0 2015-07-11 01:38 /hdfs1-r-xr-xr-x - bigred bigred 0 2015-07-11 01:38 /hdfs2
$ hdfs dfs -ls /hdfs1Found 1 itemsdrwxr-xr-x - bigred supergroup 0 2015-07-11 01:41 /hdfs1/abc
$ hdfs dfs -mkdir /hdfs2/xyz