edureka vm_ updated

Big Data and Hadoop

Version 2.0

www.edureka.co/big-data-and-hadoop

Importing Edureka VM A guide to setup Edureka VM

© Brain4ce Education Solutions Pvt. Ltd.

Big Data and Hadoop

© B r a i n 4 c e E d u c a t i o n S o l u t i o n s P v t . L t d

Page 1


Edureka VM

A guide to setup Edureka VM

Table of Contents

Install Virtual Box .................................................................................................................................... 2

Install Edureka VM ................................................................................................................................ 11

Commonly Faced Issues: ....................................................................................................................... 26

Size Compatibility Issue: ....................................................................................................................... 31

Big Data and Hadoop


Page 2


Install Virtual Box

Prerequisites:

• Minimum 4 GB RAM

• Dual Core Processor or above.

• Needed 20 GB* free Hard Disk Space to run this VM Smoothly.

* It may also run with below 20 GB but in future you may face “size compatibility" issue.

If your system does not meet the above pre-requisites, we would suggest you to use our

Remote Server.

To access our Remote Server, please refer to the document "Remote Login Using Putty -

Hadoop 2.2.0” present in LMS in the Module "Edureka VM Installation" as in the below

screenshot.

You may also refer to "Remote Login Using Putty - Hadoop 2.2.0” present in the Module

"Edureka VM Installation” of your LMS to access our remote server as in below screenshot.

FIGURE 1-0

Big Data and Hadoop


Page 3


Step 1: Download Virtual Box from below link based on your Operating System.

http://www.oracle.com/technetwork/server-storage/virtualbox/downloads/index.html Here, we have shown installation for VirtualBox-4.3.20, same steps you can follow for the updated versions. FIGURE 1-1

For Windows

For Ubuntu

For Mac OS

http://www.oracle.com/technetwork/server-storage/virtualbox/downloads/index.html

Big Data and Hadoop


Page 4


Step 2: Run the setup.

FIGURE 1-2

Step 3: Click “Next”.

FIGURE 1-3

Big Data and Hadoop


Page 5


Step 4: Select the way you want your features to be installed and click “Next”. You can also

change the location as per your will.

FIGURE 1-4

Big Data and Hadoop


Page 6


Step 5: Choose all the options and click “Next”.

FIGURE 1-5

Big Data and Hadoop


Page 7


Step 6: Click “Yes” to install VM Virtual Box 4.3.20

FIGURE 1-6

Big Data and Hadoop


Page 8


Step 7: Click “Install” to begin the installation.

FIGURE 1-7

FIGURE 1-7.1

Big Data and Hadoop


Page 9


Step 8: Click “Install” on security popup.

FIGURE 1-8

FIGURE 1-8.1

With this screen, your Oracle VM Virtual Box Manager has been downloaded and

installed successfully.

Big Data and Hadoop


Page 10


Note: If you unable to install Virtual Box on Windows, install VMware Player

which will serve the same purpose.

Big Data and Hadoop


Page 11


Install Edureka VM

Step 1: Download Edureka VM from- http://share.edureka.co/pydio/data/public/hadoop

Note: The file size of Edureka VM is 4.5 GB.

1. If you are not able to download the complete file because of internet speed, please refer the below

link for the Split files of Edureka VM.

https://edureka.wistia.com/medias/f5k5ibsucm/download?media_file_id=48883291

2. We suggest you to use the Download Manager while downloading Edureka VM to avoid any

network issues that may occur. You can download it from

http://www.speedbit.com/dap/download/ for different platforms which is an open source tool.

3. By default the Virtual Box is installed on the C Drive, in case the C Drive has insufficient

space and you have free space (20 GB) in any other drive, then to refer the further steps

Click Here

Step 2: On Import Virtual Appliance box click on the file menu to import Open Virtualization

format file (.ova) downloaded. Go to “File” menu of Virtual Box Manager and click on “Import Appliance”. FIGURE 2-1

Note: If you are not getting File option, please make sure the virtual box is in full screen mode. FIGURE 2-2

http://share.edureka.co/pydio/data/public/hadoop

https://edureka.wistia.com/medias/f5k5ibsucm/download?media_file_id=48883291

http://www.speedbit.com/dap/download/

Big Data and Hadoop


Page 12


Step 3: Select “Edureka_VM” and click on “Open”.

FIGURE 2-3

Select the location where you

have Edureka_VM.ova file

downloaded

Big Data and Hadoop


Page 13


Step 4: After selecting the .ova file click on “Next”.

FIGURE 2-4

Big Data and Hadoop


Page 14


Step 5: Click “Import” on Appliance settings box.

FIGURE 2-5

Big Data and Hadoop


Page 15


Note: After importing the .ova file in your virtual box, check the settings of virtual box.

1) Refer the screen shot below:

At bottom, if you are getting invalid setting detected, make changes in the base memory.

The cursor range should be within the limit of green line.

Note: Assign around 25-35% RAM to your virtual box of total RAM, not more than that.

Big Data and Hadoop


Page 16


2) Check the network settings:

Check adapter 1:

Big Data and Hadoop


Page 17


Check adapter 2:

Click OK and try to start the VM.

Note: If you face the below error:

Make change in both adapter as NAT.

Here, we have imported the Edureka VM successfully

and changed the needed settings!!!

Big Data and Hadoop


Page 18


Step 6: Once it got imported, you find the below image. Select “Edureka_VM” and Click”

Start”.

FIGURE 2-6

Step 7: If you get error like below, Click on “Change Network Settings”

FIGURE 2-7

Big Data and Hadoop


Page 19


Step 8: Don’t do any changes, just click “OK”

FIGURE 2-8

Step 9: Edureka VM will start on Oracle VM Virtual Box. You will have to write edureka on

password field.

FIGURE 2-9

Note: Oozie is a dummy user. There is no configuration done in that user. Password for

Oozie User is oozie

Big Data and Hadoop


Page 20


Step 10: The VM will open. On Desktop you will find LMS directory and readme file, please

go them. LMS directory has all the practical files and codes, readme file gives the information

about the VM.

FIGURE 2-10

Step 11: Open terminal and Check your hostname in terminal, and it should be in host file.

If it is not there, follow the below steps:

First Check the hostname: In my case --> localhost.locadomain

Open the host name file: (Enter password, if asked)

Note: If your host name is already in host file, close the file otherwise please add hostname

at the last as mentioned in IMAGE below:

Big Data and Hadoop


Page 21


(In my case, hostname is already there)

Note: Before you start working with Edureka VM, check if all daemons are running or not,

by using below command:

sudo jps

Output must contain:

If any of the above is missing, try following commands:

sudo service hadoop-master stop

sudo service hadoop-master start

hadoop dfsadmin -safemode leave

sudo jps

Note: Please type the command in terminal, don't copy it. It may take hidden symbols.

Big Data and Hadoop


Page 22


Note: If you have installed VMWare Player on your machine, please find the below steps to

import the Edureka VM.

Step 12: To import the Edureka VM, start the VMPlayer and click on Open a Virtual

Machine as shown in the below image

FIGURE 2-12

Big Data and Hadoop


Page 23


Step 13: Select the location where you have ova file of Edureka VM and click on open

FIGURE 2-13

Step 14: Select the location where you have ova file of Edureka VM and click on open

FIGURE 2-14

Big Data and Hadoop


Page 24


Step 15: You will find the below screen

FIGURE 2-15

Step 16: If you are receiving the below message please click on retry

FIGURE 2-16

Big Data and Hadoop


Page 25


FIGURE 2-17

Big Data and Hadoop


Page 26


Commonly Faced Issues:

1. If you get Intel VT-x or AMD-v issue, follow the steps in the document present in below link. https://edureka.wistia.com/medias/0hliot0nh5/download?media_file_id=46964037

FIGURE 1

2. https://edureka.wistia.com/medias/0hliot0nh5 3. If you get Intel VT-x or AMD-v issue , follow the steps in the document present in below

link. https://edureka.wistia.com/medias/0hliot0nh5

FIGURE 3

4. When you are trying to access HDFS, you get “NameNode is in SafeMode” , just like below

snapshot.

https://edureka.wistia.com/medias/0hliot0nh5/download?media_file_id=46964037

https://edureka.wistia.com/medias/0hliot0nh5

https://edureka.wistia.com/medias/0hliot0nh5

Big Data and Hadoop


Page 27


2. When you are trying to access HDFS, you may get “Name node is in SafeMode”, just like below

snapshot.

FIGURE 2

Solution: Go to terminal and give the command “ hadoop dfsadmin -safemode leave “ . Now

go and check your HDFS.

Big Data and Hadoop


Page 28


3. Command: oozie job -oozie http://localhost:11000/oozie -config

/home/edureka/Desktop/LMS/Oozie/WordCountTest/job.properties -run

Error: E0501 : E0501: Could not perform authorization operation, User: edureka is not

allowed to impersonate edureka

Solution: Firstly, stop oozie if it’s running.

Command: cd /usr/lib/oozie-4.0.0/

Command: ./bin/oozie-stop.sh

Three changes needs to be done.

Change 1

Edit hadoop’s core-site.xml

Command: sudo gedit /usr/lib/hadoop-2.2.0/etc/hadoop/core-site.xml

Remove oozie and put edureka as mentioned in below document, save the file and close it.

Restart the cluster.

Command: sudo service hadoop-master stop

Command: sudo service hadoop-master start

Command: hadoop dfsadmin -safemode leave

Big Data and Hadoop


Page 29


Change 2

Edit your job.properties and workflow.xml files. Use jobTracker port as 8032 in both the files and

oozie.wf.application.path as ${nameNode}/WordCountTest as mentioned in below snapshots.

Command: sudo gedit Desktop/LMS/Oozie/WordCountTest/job.properties

Command: sudo gedit Desktop/LMS/Oozie/WordCountTest/workflow.xml

Now you need to transfer the WordCountTest directory on hdfs ( / ).

Command: hadoop dfs -put Desktop/LMS/Oozie/WordCountTest /

Change 3

Giving permissions to Oozie directory.

Big Data and Hadoop


Page 30


Command: sudo chmod -R 777 /usr/lib/oozie-4.0.0

Command: sudo chown -R edureka /usr/lib/oozie-4.0.0

Now change the directory to Oozie and start it.

Command: cd /usr/lib/oozie-4.0.0/

Command: ./bin/oozie-start.sh

Run the oozie command.

Command: oozie job -oozie http://localhost:11000/oozie -config

/home/edureka/Desktop/LMS/Oozie/WordCountTest/job.properties -run

Big Data and Hadoop


Page 31


Size Compatibility Issue: To run the Edureka image, it needs 20 GB free space.

If you are not having enough space in C drive (where you have installed virtual box), then

while importing the Edureka_VM image, please follow the following procedure.

Big Data and Hadoop


Page 32


Since, you are not having enough space in C Drive, then you need to create a new folder in

another Drive.

Here, I have created Edureka folder in D drive and paste the path as mentioned, don’t remove

the last file name.

D:\Edureka\EdurekaVM_32-disk1.vmdk

Big Data and Hadoop


Page 33


Click Here to continue with next step

edureka vm_ updated

Documents

t i o n s o

u t i o n s p v t

t d page

c e e d u c

hadoop b r

putty hadoop

hadoop version

vm virtual box