Heimdall Data AccessPlatform Installation

and Setup Guide

Heimdall Data Access Platform Installation and Setup Guide

Heimdall Data Access Platform Installation and Setup Guide

iii

1. General Information 11.1. Overview of the Heimdall Data Access Platform (HDAP) ................................................. 1

2. Installation 32.1. Overview ...................................................................................................................... 32.2. System Requirements ................................................................................................... 32.3. Windows Installation ..................................................................................................... 32.4. Non-Windows Install ..................................................................................................... 62.5. Changing default port .................................................................................................... 7

3. Using the Sample Traffic Generator 93.1. Starting the traffic generator .......................................................................................... 9

4. Application Installation 114.1. General Installation ..................................................................................................... 114.2. Cold Fusion Configuration ........................................................................................... 12

5. Performance Analysis 155.1. Baseline Performance Measurement ............................................................................ 155.2. Benchmarking Heimdall ............................................................................................... 15

6. Support 176.1. Contacts ..................................................................................................................... 17

iv

Chapter 1.

1

General Information

1.1. Overview of the Heimdall Data Access Platform (HDAP)HDAP is composed of two pieces-the HDAP Driver and the HDAP Central Server. The driver can beclassified as either a type 3 (middleware) or type 4 (native Java) driver, depending on how you viewthe definition. Unlike most type 3 drivers, queries are not implicitly translated by the driver or passed toanother device. Instead, the Heimdall Driver leverages existing vendor JDBC drivers executing on theclient system to access the database, adding very little latency. All features implemented by the HDAPexecute on the same system as the application, providing scalability beyond what other middlewaresolutions provide.

The central server component operates as a control plane for one or more application servers leverag-ing the driver, and provides configuration management, logging, and failover logic for the driver com-ponent.

A brief list of the major supported features is below:

• Vendor and protocol neutral support for all features

• Provides caching both in-JVM and via external Grid cache interfaces (Memcached, Redis, EHcache,Hazelcast, and others)

• Regular expression and table aware policy configuration for caching, providing easy yet reliablecache behavior

• High availability failover for databases in simple or complex topologies including active, standby,and read-only database servers

• Read/write split of traffic to read-only database servers

• Policy-based redirection of queries to alternate data sources based on regular expressions for opti-mization and/or sharding

• Connection pooling for applications that don't include connection pooling, or have inefficient connec-tion pooling logic

• Query transformation for quickly resolving poorly written queries

• Provides run-time reconfiguration of data sources for maintenance activities

• Log SQL queries & JDBC methods, selectively or globally, for performance analysis or postmortemproblem analysis

• Query analysis for "one-click" caching based on the best cache candidates

• Access control for users, including subnet and host based access

2

Chapter 2.

3

Installation

2.1. OverviewThe free evaluation version of HDAP is a fully functioning system with the following constraints:

• Only 8 cores of processing power can be managed by the server at a given time

• 30 day license from the time of install

If a less constrained license is needed to properly evaluate the HDAP, please contact Heimdall sup-port at support@heimdalldata.com to request a license that meets your requirements.

2.2. System RequirementsThe HDAP requires a JRE of 1.6 or higher and requires at least 1GB-2GB of free memory for the serv-er component, depending on system load. The driver component has a configurable RAM cache, andwill need about 200MB of additional heap space in addition to the configured cache size. It is alsohighly recommended that anti-virus tools be disabled during install, as in particular when installing aprivate JRE, some libraries are blocked from being installed by them.

2.3. Windows InstallationHDAP can be downloaded, installed, tested, and configured for your environment in less than 5 min-utes. Here are the easy steps:

1. Downloading - First, download the HDAP (Windows 64 bit) installer package to your computerfrom http://download.heimdalldata.com/downloads/heimdalldemo.msi.

2. Installation - The HDAP installation package contains the driver, the central server, an optionalprivate JRE, grid API support libraries, and a sample traffic generator. A default install will includethe following options:

Chapter 2. Installation

4

Custom Install Selected With Feature Tree Expanded, and Default Options

Each feature is described below:

A. Heimdall Server

The central server component necessary to manage the Heimdall driver component. This onlyneeds to be installed on a single machine, and can be used from many machines (license per-mitting).

B. Heimdall Service

If you wish to install the Heimdall Server as a system service under windows (recommended),please leave this selected. On install, the service will be started automatically.

C. Private JRE

Only necessary to have selected if no system-wide JRE is available. This should automaticallybe selected for installation if this condition applies.

D. Example Configuration

If installed, the Heimdall Server component will startup with a configuration already in place.For evaluation purposes, this is recommended. This configuration can be modified post-install.

E. Driver Library

This can be installed with the Example Configuration or separately, but is recommended, asit simplifies the configuration of database access through Heimdall. The driver library can beadded to or modified after installation.

F. Heimdall Driver

Windows Installation

5

This component only needs to be installed on a system that an application will be run. TheHeimdall Driver install location can be modified to point to the library directory of an applicationdirectly to simplify the install process.

G. Grid API Libraries

When installed, a lib directory will be created under the install location of the Heimdall Driver,with supported grid cache API libraries. Usually, if used in an application, this is not neces-sary, as they will already be present in the classpath, and will be used automatically by Heim-dall. Generally, this is used for testing the demo application with grid interfaces.

H. Traffic Generator Example

To provide a simple set of traffic for demonstrating or understanding features, this feature in-stalls into the traffic directory under the Heimdall Server install directory by default.

3. Service Startup - If the Heimdall Service component was selected, the Heimdall service shouldhave been started automatically, and the console should have been opened in the default webbrowser. You can start configuration immediately. To start and stop the service, use the Servicesmanager:

4. Manual Startup - If the service installation is not used, the server component can be started man-ually by navigating to the install location (default c:\Program Files\Heimdall) and executing thescript "serverstart.bat" as administrator:

Chapter 2. Installation

6

5. Accessing the Central Manager - On install, if the service component was installed, you shouldautomatically be directed to the console at http://localhost:8087/. This same URL can be used toopen the console at any time.

6. Getting Configuration Help - Each tab has a "help" option at the upper right corner of the tab thatprovides access to online documentation. Please refer to this documentation for detailed. As anexample, on the Virtual Database tab, you would use the highlighted button as shown below:

2.4. Non-Windows Install1. Downloading - First, download the OS independent install package at http://

download.heimdalldata.com/downloads/heimdalldemo.tgz. The only host software re-quirement is that the system supports a JRE 1.6 or higher.

Changing default port

7

2. Installation - The HDAP installation package contains the driver, the central server, grid API sup-port libraries, and a sample traffic generator. Decompress the archive into a directory (/opt is sug-gested).

3. Service startup - As different operating systems have different requirements for startup, a simplescript is provided that can be called from either an init script, from rc.local, or by another startupmechanism. The script is "heimdallserver.sh" and will accept the options for "start" or "stop" in or-der to start or stop the Heimdall server.

2.5. Changing default portThe default port used by the Heimdall Server is tcp port 8087. In order to change the port, simply cre-ate a file named "application.properties" in the install directory, and add a line like:

server.port = X

Where X is the desired port number.

8

Chapter 3.

9

Using the Sample Traffic Generator

3.1. Starting the traffic generatorThe included traffic generator source is installed by default into the directory c:\Program Files\Heim-dall\traffic, and can be run by executing the runme.bat also in the same directory. On the initial run, thesource will need to be compiled by a locally installed javac, which is included in the Java JDK environ-ments by default. The private JRE that may be installed to support the service component does not in-clude a javac executable. If a JDK is needed, then it can be downloaded from Oracle. Once javac isavailable, the runme.bat script can be executed as administrator to compile the source, and then to ex-ecute the result (a warning will be presented if this does not work):

Upon execution, the traffic generator will use the sample configuration that is by default installed intothe Heimdall Server, and will leverage a database available on the public Internet to allow demonstra-tion of the various features of the HDAP. Once started, the server console should provide a view suchas below on the dashboard, which can be used to confirm that traffic is being passed properly:

Chapter 3. Using the Sample Traffic Generator

10

From here, feel free to explore the features that the HDAP provides via the online help, and throughthe sample configuration that is provided.

Chapter 4.

11

Application Installation

4.1. General InstallationWhen starting the process of installing Heimdall into an application, in general, the following stepsshould be taken:

• Select an application server/client that can be restarted, and traffic driven through outside of produc-tion.

• Insure that the system selected is operating normally-this is extremely important, as often issuesarise during Heimdall install that are not related to Heimdall at all.

• Copy the Heimdall driver jar file into the classpath of the application. Normally, it is best if you findthe current JDBC connector jar file, then copy the heimdalldriver.jar into the same directory.

• Find where the application configures the JDBC class path and Java class for the the JDBC connec-tor. For example, with the Atlassian Bitbucket server product, you will find the configuration in:

/var/atlassian/application-data/bitbucket/shared/bitbucket.properties:

#>*******************************************************

#> Migrated to database at jdbc:mysql://localhost:3306/bitbuck-et?characterEncoding=utf8&useUnicode=true&sessionVariables=storage_engine%3DInnoDB

#> Updated on 2015-10-19T20:33:48.292Z

#>*******************************************************

jdbc.driver=com.mysql.jdbc.Driver

jdbc.url=jdbc:mysql://localhost:3306/bitbuck-et?characterEncoding=utf8&useUnicode=true&sessionVariables=storage_engine%3DInnoDB

jdbc.user=user

jdbc.password=password

• As another example, for Atlassian Jira, the configuration is in:

/var/atlassian/application-data/jira/dbconfig.xml:

...

<url>jdbc:mysql://localhost:3306/ji-ra?useUnicode=true&amp;characterEncoding=UTF8&amp;sessionVariables=storage_engine=InnoDB</url>

<driver-class>com.mysql.jdbc.Driver</driver-class>

<username>root</username>

Chapter 4. Application Installation

12

<password>a1k81ucuTr</password>

...

• Create a data source object with the configured JDBC driver, JDBC URL, user and password.

• Verify that the central server can connect to this data source object.

• Create the application's VDB.

• Edit the application configuration to use the JDBC URL and Class specified in the VDB configura-tion.

• If you are using authentication on your Heimdall Server, use the username and password to authen-ticate to Heimdall as the JDBC username and password, as the database username and passwordwill be configured via the data source.

Note: If you have dynamic or multiple users and passwords, and the application controls this, you canuse a JDBC query parameter of "hduser" and "hdpassword" to provide Heimdall Server credentials.

• Start the application.

• Check the application console logs for messages similar to the following, which indicates a properstartup of the Heimdall Driver.

[2015-10-20 06:06:02.344] Heimdall: Retrieving updated configuration for URL: http://local-host:8087/dbdemo

[2015-10-20 06:06:02.345] Heimdall: getting config from: http://localhost:8087/api/config/vdb/dbdemo

[2015-10-20 06:06:02.545] Heimdall: heimdall: Startup performed:com.heimdalldata.log.EventLogger@7de87192

[2015-10-20 06:06:02.546] Heimdall: Initializing logger: com.heimdalldata.log.EventLogger@7de87192

...

• All messages being generated by the Heimdall Driver will start with the format [YYYY-MM-DDHH:mm:ss.SSS] Heimall: message. If they can't be found in the application logs, it is likely that thedriver was not initialized by the application properly, and other application errors should be checked.

• Common issues with configuration:

1. The driver was not found in the application class path

2. The JDBC URL or Classname was not configured

3. Port 8087 is not open for access to the Heimdall Server

4.2. Cold Fusion ConfigurationTo install the Heimdall Driver into Cold Fusion, please follow the following steps.

1. Copy the driver into to the Cold Fusion lib directory (by default C:\ColdFusion11\cfusion\www-root\WEB-INF\lib).

2. In the Cold Fusion GUI, navigate to the Data & Services > Data Sources window:

Cold Fusion Configuration

13

3. Using the "other" data source type, add a new data source, and configure it with the appropriateHeimdall JDBC URL, but with all other settings identical to the original data source used previous-ly:

Chapter 4. Application Installation

14

Chapter 5.

15

Performance Analysis

5.1. Baseline Performance MeasurementIf you can't measure it, you can't improve it - Peter Drucker

In order to determine how effectively Heimdall is able to improve performance, it is necessary to firstcreate a reproducible performance test that can be executed many times to get statistically meaningfulperformance measurements. There are many different approaches to doing this, and tools to providethis. Some of these include:

• For web applications, "true" user performance can be measured with Dynatrace's or Catchpoint'sproducts.

• For local Web based application performance benchmarking, Jmeter or LoadRunner can be used.

• For GUI applications that rely on databases, a tool such as Cucumber (https://cucumber.io/) can beused.

For any such testing, it is important to insure that the results of the baseline are reproducible, and con-sistent between test runs. Once a baseline performance test has been run, Heimdall can be installed,but several considerations need to be made first:

• Restart the application between each test. As Java can optimize execution over time, and an ap-plication can also leak memory or cache on it's own, it is important to make sure the application isrestarted to get consistent results between test runs.

• Install the Heimdall Server component on a separate server for performance testing, as the addition-al load from log processing and memory overhead may slow down the application.

5.2. Benchmarking HeimdallOnce a reproducible baseline test has been created, the Heimdall Driver can be tested:

• Test the application with the VDB "pass-through" option first, to ensure that the installation behaveswith a similar performance profile. Some applications, if you install a custom driver, will behave dif-ferently vs. a built-in driver. Cold-Fusion for example, issues different JDBC calls for the MySQL dri-ver vs. a custom driver. The Heimdall driver has been optimized to detect this behavior, but otherapplications may also behave differently with different drivers.

• Assuming similar performance has been observed in pass-through mode, test without pass-throughenabled, but no other logging or policies in place. Again, performance should be close to identicalvs. the non-Heimdall performance profile.

• Next, test with SQL logging enabled. It is possible that this will induce a small (no more than 5-10%)reduction in database performance.

• With SQL data gathered, utilize the Analytics system to configure caching for appropriate queries.Keep in mind that not all recommendations make sense, but the recommendations can narrow downwhat policies to consider.

• Test with caching enabled, but disable logging again. This should be the normal behavior in produc-tion, although sampling can be used to provide an overview of traffic without significant performanceimpact.

Chapter 5. Performance Analysis

16

• In the event that performance is not improved, or is negatively impacted when it shouldn't be, enablemethod and result set method logging on the VDB, and test with a very low load test, and then pro-vide the logs from the logs directory (default c:\Program Files\Heimdall\logs) to Heimdall Support foranalysis.

Hints for successfully benchmarking without test artifacts:

• Monitor the system performing the test for CPU load and memory saturation. Often, a benchmarktool is actually measuring it's own ability to generate traffic, not the system under test.

• Try to avoid using virtualized environments for testing. Using dedicated hardware for clients, serversand networks will help reduce the number of variables that can impact a test.

• Keep an open mind on results, in particular when results look particularly good or bad, as either mayindicate an issue in test methodology.

• Always consider variables relating to the test environment in mind. If large numbers of connec-tions are being established and torn down, consider TCP port use and reuse. Searching on the webfor "windows ephemeral port exhaustion" will bring several topics about this common issue duringbenchmarking or heavy server load.

• Ask for help in analyzing results and the overall test setup! Heimdall Support can work with you toensure that the appropriate setup is used for reliable results.

Chapter 6.

17

Support

6.1. ContactsFor any issues, comments or suggestions, please use any of the following contacts:

E-Mail: support@heimdalldata.com

Phone: 1-866-HDDATA4 (1-866-433-2824)

Website: WWW.HEIMDALLDATA.COM/contact.html

18