How to Lay Down a New JVM

On a Big Data Cluster

By Mason Yu Jr.

In a Hadoop cluster there are many moving parts

and a cast of supporting characters. It is not as visual and spectacular

as producing an action packed Hollywood movie with a host of

A-list actors and beautiful models, however, if the director is out

for a period of time, most likely the movie to be made will suffer from

a lack of purpose and vision and certainly cant possibly win an Oscar

or two.

Working on a project with a small POC, it was discovered

that the existing JVM was over two years old. That in itself caused

a number of extremely inexplicable errors in the Hadoop system that

the only possible way to eliminate the root causes is through laying

down a new JVM. It is a job for the faint hearted, but it is an excellent

way to understand the technology stack. Big Data means parallel

distributed data computation. The corollary to this is Big Data has

distributed systems administration. What once was a monolithic

operation on a single systems container, it is routine to have dozens

of machines. For scalability of systems administration, Puppet and

Chef can automate the processes to make the Linux systems

administration far less prone for human error and accelerate the

deployment process. At time of writing of this paper, a state-of-art

deployment tool known as Ansible has been available for general

release.

The focus of this write up is to illustrate in detail

the number of steps required rather than using a admin automation

tool. The starting point of this exercise is at a zero state of entropy

where all the Linux daemons are down. This is a permissible state,

since the Linux cluster itself is a small development platform.

For a larger Linux cluster, much more coordination and a smaller

window of deployment would be necessary.

Setting the stage

5 Linux instances running Red Hat Enterprise release 6 64 bit

2 Name Nodes, 3 Data Nodes

Remainder of the Hadoop artifacts(refer to the subsequent screenshots)

Hadoop distro – HortonWorks HDP 2.3.2.0

The next page will dive right into the Linux cluster. After the

initial login and password, this is the main screen showing the cluster.

Swiss army knife

WinSCP - to be downloaded on the Windows machine

Putty – To be downloaded on the Windows machine

Command line skills on Linux

Navigational skills around Ambari

Before creating this document, the entire cluster has been brought

down to a ground zero state. This is mandatory, since the Java JVM

is the foundation for the Hadoop Big Data services and ecosystem

relies upon. This is obvious since the components are written in Java.

Over the years, Oracle has continued a long standing effort to enhance

the Java packages. In the process of doing so, some of the Java classes

and methods have been deprecated. As the authorship and participants

in the Apache Open Source group have made software changes using

the newest JRE/JDK, the users of the Hadoop system need to upgrade

their JRE/JDK systems as well.

Getting back to the main screen, many of the smaller graphs have

a n/a. This is because the various Linux daemons representing the

Hadoop services have been stopped. The next set of screenshots will

lay down the newest JVM/JRE.

The source of the JVM/JRE is none of than Oracle.

On your favorite browser, go to www.oracle.com. See below:

Next point the mouse to the top center of this screen, go to the hyperlink

called Downloads, click on it.

Now scroll down to you see the next part of this Web page:

Hover over the icon which says Java downloads, then click on it.

Now scroll down this Web page until you for the section for Java:

Hover the mouse until it is on top of the hyperlink Java SE and then click on it.

Now hover the mouse over the JDK download button, click on it.

Be sure to locate the Java SE Development Kit 8u92 !!

Now hover the mouse and click on the radio button Accept License Agreement, click on the radio button.

Notice that the radio buttons disappear and in lieu is the statement of

acceptance. We need to narrow down the scope of JDK offerings by

OS and by word size. Since the Linux cluster is a specific type of

OS, note that the target machine is Red Hat Linux Enterprise

6.5(Santiago). Run the command from the Primary Name Node

At the command prompt, enter cat /etc/redhat-release.

Getting back to the task at hand, the RHE accepts and processes rpm

files. To make sure that there is a 64 bit CPU, run from the putty

session a uname -I command. See below:

On the Windows page, locate under Java Development Kit 8u_92

section, the file Linux_64 …………….. jdk-8u92-linux-x64.rpm

Hover the mouse over this jdk version and click on it.

Depending on the type of browser, in this case it is IE 9, there is

a horizontal orange bar at the bottom. Hover the mouse over the

button named Save, then click on it. Because the Linux cluster

can only be accessed by secure VPN, any outside machine will not

be able to directly install to any of the members of the Linux cluster.

That is the reason, the Linux rpm is being saved locally.

At this juncture in time, depending on the configuration of the Windows

local machine, click on the button which says Open folder.

Notice the highlighted file – we have just downloaded the Linux rpm

to the local folder \..\Mason\Downloads.

The next task is to get the companion JRE(Java Runtime Environment).

This is a similar procedure. Go to the back page button of your browser,

it will look something similar to this . Hover the mouse over this

icon and then click on it:

Near the bottom of this page there is a prominent button which says

on top JRE. Just below it is the download button. Hover the mouse over

it and click on this button.

This last screen is déjà vu. Once again there is rich choice of JREs,

CPU widthsize, type of operating system, etc. Since we selected the

8u_92 version of the Java jdk, the version of he JRE must match.

Scroll midway down this Web page and find the section of the JREs

which are 8u_92. We need to accept the license agreement. Hover

the mouse over the radio button which has the words Access license

agreement next to it. Click to accept.

Once this is done, the Web page will now display a courteous thank you

note. Now we can select the appropriate JRE for the Linux cluster we

plan to upgrade.

Under the Download column for JRE 8u 92, look for the file entry

jre-linux-8u_92_x64.rpm. Hover the mouse over this specific file

and click on it.

Notice at the bottom of this Web page a orange horizontal bar appears

at the bottom. Hover the mouse over the button which says save, then

click on it.

Just to make sure we have the second Linux rpm file, hover the mouse

over the bottom of this page and click on the middle button Open folder.

Now we have to switch gears and insure the Linux cluster has

Java environment variables set up properly. Since a putty session

is already open on one of the Linux instances, we will now turn our

attention to this.

There are several variables which need to be validated and changed –

JRE_HOME, JAVA_HOME and CLASSPATH. This must be done

on all 5 Linux instances. The profile bash script needs to be adjusted.

Also, to execute these commands and to insure that the Java artifacts

are installed correctly, root privileges are mandatory.

We will focus on the primary Name Node, login in to

Linux instance with the Windows Putty utility, place the Class C IP 4

address in the text box, hit enter:

Hove the mouse over the button labeled Open at the bottom of this

utility, then click enter on the screen:

There not much to this, your Linux administrator will have set this

access to the Linux environment. Type in your login id. Then hit enter.

As mentioned earlier, the JVM changes must be executed with root

privileges. From the prompt, type sudo su - Then hit the enter key.

Despite several attempts of getting the right password, the third time

worked. The key thing to note is that before the prompt, there is a

[root], indicating this login is now signed in as root.

To pretty this session, Putty has a nice feature. Move the mouse along

the top of the Putty window bar. Next right click the mouse and a

drop down menu appears. Scroll past half way down, there will be an

option Change Settings… Left click on this entry. The resultant

window look like this:

It looks very familiar. This is original screen in Putty. Under Category,

look past halfway down this column for Colours. Click on this entry.

The plan is to have a yellow background with dark blue letters.

First move the mouse to the top entry in the box entitled Select a colour

to adjust. With the mouse click on the entry which says Default

foreground.

You will see a 187 in 3 textboxes, ignore these and click on the

button labeled Modify.

Note that the color rectangle default is a light gray above the

word Color/Solid. Now move the mouse and hover over the

color closest to blue and click on the selection.

Now hover the mouse and click on the button OK.

The only thing left is to change the background of the Putty screen.

Move the mouse to the entry Default background, click this entry.

We will pick yellow, hover the mouse over the button Modify,

then click on this entry.

Now hover the mouse over the color square yellow; click on this

color square.

Notice that the rectangle labeled Color/Solid has now changed to yellow.

Now close out this dialog box – hover the mouse over the button OK,

click on it.

Last step, hover the mouse over the button entitled Apply, then click on

it.

Now we are working with bright lights.

Because Linux comes in a variety of distros as the number of

scripting languages, it important to know the specific distro – the

syntax varies ever so slightly, so commands which may look convincing

on paper, may not necessarily work for the technology stack.

Typically a man page ought to illuminate any discrepancies.

Just type man the name of the command, e.g., man iostat.

From the command line of the Putty window, type clear and then hit

enter. Next, we need to find out where is the actual Java jdk is parked.

Every system has its own quirks. Execute a command which java.

Execute another command whereis java. In both cases, hit the

enter key.

There are several more commands to help piece the puzzle here. Since

there was another group which had development access to this Linux

cluster, we need to find out where Java resides and whether there are

symbolic links. From the command line, issue/type the following:

ls -lah /usr/bin/java. /usr/bin/java is the resultant response from the

previous Linux command which java.

Notice that /usr/bin/java points to /etc/alternatives/java. We need a little

more gumshoe effort. Now type cd /etc/alternatives, hit enter. Then

type ls -l, hit enter.

The symbolic link highlighted shows java /opt/jdk1.8.0_5/bin/java.

Now that we know the lay of the land, we will transport the Linux rpm

binary of the latest Java JDK from the local Windows machine into the

target Linux instance 10.20.40.85. Click on the icon for WinSCP.

Now type in the Linux host IP address, the designated login and

password assigned to you. Hove the mouse over the button entitled

Login, click on it.

What you see next is the main screen of WinSCP. This is undoubtedly

one of the best tools for cross platform work between Windows and

Linux. Others tout FileZilla, so it is really a matter of practicality and

features. Refer to the next screen.

On the left panel is the local Windows file system and on the right is the

Linux instance. We will make the screen is little more colorful.

Navigate to the top, locate the word Options, click on it so that the drop

down list appears, scroll to the near the bottom of this list, select the

option Colors, scroll the mouse to the far right until the color palette

appears, select the small color square and then click on top of it.

To assist locating the two .rpm files already downloaded from the Oracle

Web site, hover the mouse near the top of the WinSCP window, locate

the word Options, click on it to open the dropdown list. Locate the

option Local Panel, move the mouse to the far right until another drop

down list appears, hover the mouse until it is on top of the entry

Tree, then click on this. Notice the left panel is now split into two

sub panes.

Now locate the rpm file for the jdk. In the lower right sub pane,

highlight this entry.

Make sure that the remote(Linux) directory is where the new jdk is to be

parked. In this exercise, the new jdk will be parked under the /opt

directory on the remote Linux instance. Now hover the mouse over the

local jdk……rpm file entry. Next, right lick on this entry and a drop

down list will appear. Look for the option Upload. Then click on it.

In case the permissions becomes a roadblock, go back to the remote

machine and execute from the command line, chmod 755 /opt -R.

Replace /opt with the name of the remote directory of where the jdk

…rpm will be landing. If this does not work, navigate to the /tmp

directory. Then Upload the rpm file.

Once the data transfer is completed successfully, locate the file

Jdk….rpm on the remote directory /tmp. Refer to the screen on the

next page. Notice the bottom most entry on the right pane. That is

the jdk we just uploaded !!!

Next, copy the rpm file from /tmp to /opt or to the target directory

of where the jdk needs to be. Execute from the command line,

cp /tmp/*.rpm /opt. This is assuming there is no other .rpm file.

Then execute a cd /opt, then run a ls -l. The jdk…prm will be there

as planned.

Next from the command line, execute rpm -ivh jdk.1.8.0_92.rpm.

Where is the newest jdk ??? Though the command was executed

in the /opt directory, the installation unpacked all the binaries

into /usr/java. We ran whereis java and which java earlier, that

gave us a clue where it would be.

Next we need to copy everything in this latest jdk from /user/java to

/opt. Execute the command cp -avr jdk1.8.0_92 /opt. The screen

below is the results of the tail end of this execution.

Go back to the WinSCP to validate the jdk folder transfer.

We have just installed the latest jdk on the remote Linux instance.

We now have to fix the Java environment variables. Going back to

active WinSCP session, navigate to the remote panel(right side),

click on the left most forward slash. It is located just above the

Name column. The file listing will exhibit all the folders in the /

directory. Locate the etc folder and click on it.

Scroll down and look for either file /etc/profile or /etc/bash.bashrc.

Exploring both files in this instance, the bash.bashrc will be changed.

Be sure to make a backup copy. Execute the following commands,

cd /etc, then cp bash.bashrc bash.bashrc.MMDDYYYY. Then run

a command, ls -l | grep bash.

Now switch back to WinSCP, click on the entry bash.bashrc.

Notice it has all the well Java environment variables we need to change.

So the JAVA_HOME, JRE_HOME and CLASSPATH have been

changed. Don’t forget, click on the little floppy disk icon in the NW

corner of this windows to save the changes made to this file.

Go to the Putty screen and validate the text line changes. If the edits

can not be made in WinSCP, drop back to vi, then use cat to validate

changes.

Earlier in the paper, we have to look for /etc/alternatives for

a symbolic link. The existing symbolic link points to the old JVM

_05. We need to have the java symbolic link point to the new

one we just installed in the /opt directory.

First execute a ls -l java to verify the symbolic link.

Next run a unlink java command. This the command to remove the

actual link. Run another ls -l java command to make sure the link is

removed. Next we will create a new java link. Execute a

ln -s /opt/jdk1.8.0_92/bin/java java. This command creates a new

symbolic link to the newly installed jdk accomplished earlier in this

endeavor. Last, run another ls -l java to insure the new symbolic link is

there. See the screenshot below.