show104 - buried treasure: finding the hidden gold in lotus notes data

© 2013 IBM Corporation

SHOW104 Buried treasure: Finding the Hidden Gold in Lotus Notes DataMark Myers | London Developer CoopJulian Robichaux | panagenda


Agenda

Overview

Working with Feeds

Integrating with IBM Connections

ODBC, JDBC, and Reporting

Generating Files on the Domino Server

Caveats, Considerations, and Data Scrubbing

Where to Get More Information


Who Are Mark & Julian?

Julian– Java/Eclipse application developer at panagenda (panagenda.com)– Developer since before Justin Bieber was born– Notes/Domino since version 4.1– Speaker at 7x Lotuspheres, various LUGs and View conferences– Twitter: @jrobichaux, Blog: nsftools.com

Mark– Member of the London Developer Co-op (londc.com)– Developer from a support background– 12+ years on Domino, 15+ years in IT– Speaker at 2x Lotuspheres, 3x UKLUGs, 1x ILUG– Twitter: @stickfight, Skype: Stickfight, Blog: stickfight.co.uk


Why Are We Here?

You have years and years of data stored in your Notes databases– Valuable– Critical?– Historically important

Often, that data needs to be accessed from other places

– Reports– Intranets/portals– Non-IBM systems

What are your options?

As it turns out, you have a lot of options…


Tools We Used

IBM® Lotus® Notes® version 8.5.3

IBM Lotus Domino® version 8.5.3

Microsoft® Windows® 7

IBM Connections V4.0

Various other software tools mentioned throughout this presentation

Most code and techniques we talk about should be applicable to other versions of Notes/Domino/Windows too


Old and New

This presentation is an eclectic mix of Old and New, borne out of many many years of Domino programming experience

Solutions are based on Ease, Speed and Reliability rather than the latest coding fashion

If a solution seems old (more than 2 year old technology) it is because: – Some implementations we see are still being done wrong (or not done at all)– Some people are still on old versions of software– Sometimes the “old” way is still the best way to do things– Everyone is at a different experience level; maybe you’ve never seen some of this before– This stuff still works


About the Sample Database

This year’s sample dataset is… Sea Ducks!

The Atlantic Flyway Sea Duck Survey, conducted by the U.S. Fish and Wildlife Service, was established in 1991

to record sea duck numbers using near shore (within 700 m of shore) habitats from Cape Breton, Nova Scotia to

Jacksonville, Florida.

https://migbirdapps.fws.gov/mbdc/databases/afsos/disclaimerafsos.html


About the Sample Database

Public domain data and photos from the US Fish and Wildlife service– See the “About this Database” page in the sample database for information, disclaimers,

and links

A little over 10,000 lines of data (i.e. – individual documents in a view)

Various means of grouping and data retrieval– By year– By state– By duck type

Some documents also have image files stored in rich text fields

So… potentially similar to data you already have


Agenda

Overview

Working with Feeds







What’s a Feed?

Structured data– 0-to-N items

Parseable format

Retrievable via a link or some other automated method

Ideally (but not necessarily) semantic– Semantic markup assigns “meaning” to data

http://en.wikipedia.org/wiki/Microformat


JSON vs. XML


ReadViewEntries

Replace ?OpenView with ?ReadViewEntries in the URL of any Domino view on the web to see the contents as XML

– Since Domino 7


ReadViewEntries

Some optional parameters:– CollapseView / ExpandView– NavigateReverse=1 – lists documents in reverse order, starting with the last document– Start=n – indicates which document number to start with– StartKey= / EndKey= – indicates which key in a sorted view to start and/or end with– RestrictToCategory= – indicates which category to display in a single-category view

Count=nn – Very important if you want to get ALL the docs in a view– Default is 30, or whatever is specified by “Default lines per view page” on the Domino

Web Engine tab of the server (or web site) document– Default maximum is 1000, or whatever is specified by “Maximum lines per view page”

OutputFormat=JSON – Displays data as JSON instead of XML

A few other parameters too: check Domino Designer help


ReadViewEntries JSON Format


ReadViewEntries XML Format


Notes on Using ReadViewEntries

Maximum View Count settings on the server doc might prevent you from seeing all the data in the view

You can save the query and refresh the data on demand

Categorized views get messy, flat views are better

If you need to login, make sure the server is using Basic Authentication (at least for that URL)

– Not strictly necessary, but much easier from a programmatic standpoint


Overriding Session Authentication

If your Domino server uses session authentication (most do), there is an option to override this and use basic authentication for specific URLs

Since Domino 7– http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/

com.ibm.help.domino.admin.doc/DOC/H_OVERRIDING_SESSION_AUTHENTICATION_8847_STEPS.html

Must be using Web Site documents for web server access– “Load Internet configurations from Server\Internet Sites documents” enabled on Basics

tab of server document– http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/

com.ibm.help.domino.admin.doc/DOC/H_R5_WEB_SITE_TO_RNEXT_OVER.html



Open the “Web Site” document for your server



• The URL pattern can use * for wildcards (? is treated as a literal character).

• If you don’t a wildcard at the beginning and end, the rule might not work the way you expect.



Optional parameter on the Web Site document to auto-generate a session cookie when Basic Authentication is used in an override rule


Classic Feed: ICalendar

Still the best format for dealing with event based data

Near universally supported

Can be done without 3rd party libraries



We just need a view.


Classic feed: ICalendar

Most ICalendar clients tend to be picky on carriage returns so you need to be careful



ICalendar date time is in the format yyyyMMddTHHmmssZ


Classic feed: ICalendar

Then insert the view in a form with the text that wraps the ICalendar Vevents, this text details time zones and such for the calendar as a whole


Classic feed: Web Services

The De facto standard for inter-system communication

Still seen as a silver bullet by many managers

See a new web service as an opportunity to re-define how your data is seen

Complex if done badly, simple with a few tricks or the right software


Classic Feeds: Web Services

Best Tools– Lotus Domino server!

– Liquid XML Studio ( http://www.liquid-technologies.com/xml-studio.aspx )• Brill visual designer• Powerful but easy to use• Costs a lot from a stingy developer’s point a lot (but worth it)

– Soap UI ( http://www.soapui.org )• The simplest way for humans to interact with and test raw web services• Free for the Lite version


Classic Feeds: Web Services


Custom Feeds: Decisions to Make

Data format– XML, JSON, CSV, something else…

Data structure– RSS, Atom, something else…– Standard structures are widely supported– Custom structures allow more flexibility

Static or dynamic– Caching and performance issues– User-specific information?

Do you control the clients?– If not, it might be hard to make changes later


Atom and RSS

Standard XML formats for document-based data

RSS is simpler to produce and parse– Fewer options– Easy to understand– Great for syndication of news, blogs, and similar

Atom is more flexible– Easily extensible for data customization– Read AND write data, if you need to publish– Still primarily for documents, rather than raw spreadsheet-type data

Almost any software that calls itself a Feed Reader can consume either type of feed with no problem


Feed Library vs. Build-Your-Own

XML and RSS/Atom is easy to break– Character encoding issues– Illegal characters– Unclosed tags– Improperly formatted dates– Missing required elements

If you write “raw” XML to a text string for output, test often and add plenty of checks for poorly formatted data

Feed libraries take care of the heavy lifting for you– Built-in validation– No need to study the specification documents– Probably slower and more memory-intensive– Usually several megabytes of required code libraries either stored in the database or

copied to the server


Example: RSS Feed Generator Database

Template included with Domino 7

Easy to use for quick feeds:– Create a new database on the server using the “RSS Generator” template– Fill out a form to point to a view– Point the users to the database URL

Reuse the code in your own applications



Unfortunately, the <enclosure> item has a hard-coded data type, so it’s only good for docs where all the attachments are the same type


Example: Java Agent Atom Feed

A few different Java feed parsing libraries to choose from:– ROME – http://rometools.org – Apache Abdera – http://abdera.apache.org – Apache Wink – http://incubator.apache.org/wink

All require several additional libraries– Apache Commons libraries– Java StAX support– Etc.

Handle HTTP connections as well as parsing– Authentication too

An excellent test client:– DEV HTTP Client (Chrome Plugin)https://chrome.google.com/webstore/detail/

aejoelaoggembcahagimdiliamlcdmfm


Example: Java Agent Get Atom Feed


<?xml version="1.0" encoding="utf-8"?><entry xmlns="http://www.w3.org/2005/Atom"><author><name>I Am Funny</name></author><title type="text">Great Duck Joke</title><content type="text">Q: What has webbed feet and fangs? A: Count Duckula</content></entry>

Example: Java Agent Submit Back to Atom Feed

Most Atom services (particularly IBM) give examples of what XML they want rather than examples of the code you should use, so we will do the same

This is a IBM Connections Blog entry in raw XML:



An amazingly useful line when you are debugging or testing as it outputs the XML so you can check it against what you should be sending



Getting this URL is often the hardest part of dealing with Atom


Example: Useful Extra LinesAdd “registerTrustManager”

and you can connect to SSL Ports

Nice simple login, you just need the user name, password and the root domain you are logging on to


Agents vs. XAgents

Java agents are not efficient when using large JAR files attached to agents or script libraries

– They are reloaded into memory each time the agent runs– Beware of memory problems

Copying the JAR files to the jvm/lib/ext directory on the Domino server keeps them in memory

– If your admin allows it– Have to do this with every server the code might run on

An alternative is XAgents: XPages written so they run as agents– No UI, except for print statements (“write”, actually)– JARs and Java code attached directly to the database– Much better caching of JAR files– http://www.wissel.net/blog/d6plinks/shwl-7mgfbn – http://www.mindoo.com/web/blog.nsf/dx/17.07.2011101855KLEBRW.htm


Caveats

Malformed JSON and XML

Character encoding issues

Multi-value fields can be tricky

Memory issues when using large Java libraries– Or poorly written Java code

Make sure you use the right Content-Type HTTP header when you send data– JSON should be “application/json”– XML should be “application/xml” (preferred) or “text/xml”


Agenda

Overview

Working with Feeds







Integrating with IBM connections

2 basic forms of integration– Submitting existing data directly into Connections (as demonstrated by the previous

Atom examples)

– Read data from Domino and displaying it in Connections via a widget


Connections Caveats

This is not a Connections session

The methods shown in this session are one of many methods of integration rather than specific best practices

– Niklas Heidloff (http://heidloff.net) and Mark Leusink (http://linqed.eu) have some good examples

– These methods use as few toolkits such as Social Business toolkit/Social business SDK, social enabler as possible.

Real life has taught that the best integration environment to build is one where you can get all data if required (Gather all data using service accounts) then compile a result and display to the user.

See the related sessions at the end of this presentations for more details and the best practice sessions.



Before we start we need to get a database ready to call Java from XPages

Move to Package Explorer



In The WEB-INF directory

Right click and create a new folder



Call it “src” in lower case



Next add this folder to the projects build path

Right click on the root of the project and select “Properties”


Integrating with IBM ConnectionsOn the “Source” tab click “Add folder” and add the folder you have created


Integrating with IBM ConnectionsYou will see the Folder has moved

Right click and select “New” → “Other”

Then Select “Java” → “Class”


Create the new Java Class



OK, Java class created, lets get it some libraries



If you are not allowed to use the EXT directory on the server, the you will need to create a new “lib” folder



Update (or ask your administrator to) the java.policy file on your server to contain the following


grant {

permission java.lang.RuntimePermission "getClassLoader";

};


Now we will add the Atom client code function

You need to pass the session into this function as it can’t get its own

If you need to you can login here (as described previously)



Create a New Script Library



Set it as a Normal “Server JavaScript” Library



Pass the “sessionAsSigner” here to give yourself administrator rights

Create a SSJS function to call your Java and pass the results back



Integrating with IBM Connections – Pulling it together

Create a Custom Control


Give it a Name



Now you can add a repeat control that calls the SSJS function, which in turn calls the Java


The Rest of the surrounding boring code is in the demonstration db



Create a New Xpage Library



Add the duckstates_cc custom control


Now that you have an XPage, you need to present it to Connections as a Open Social widget the easiest way of doing this is with a simple domino page.




Features like “dynamic-height” are harder to get working using this method (rather than a embedding the widget in connections) but if you have a static sized widget it is the simplest.


And add it to your Connections installation



Agenda

Overview

Working with Feeds







SQL Queries of your Notes Data?

Lotus Notes is NOT a relational database– You knew that, right?

IBM has a NotesSQL driver– Free download, easy install– Treats Notes Views like SQL tables– Has been updated for Notes 8.5


From the Notes Client: NotesSQL

Free tool from IBM

Windows-only (make sure you have the latest version on Windows 7)

Set up on each user’s machine (install plus DSNs)

Notes client must be installed (version 6.0 or higher)

Easy integration with reporting tools like Crystal Reports, MS Excel, and MS Access


Setting up NotesSQL

Download from http://www.ibm.com/developerworks/lotus/products/notesdomino/notessql

– Help files are in C:\NotesSQL\docs after install

After install, the NotesSQL directory and the Notes program directory MUST be in the Windows PATH

– You get vague error messages if they’re not

Create an ODBC DSN to point to a database


Example of Connecting with MS Excel


Optimizing Your Data

You will almost certainly want to create one or more “reporting” views– Many columns of data– No categorization– Minimal sorting for better lookup performance– Reasonable column names (check the “Programmatic Access” name for computed

column data)

If you’re reporting on rich text data, modify the DSN to allow more than 512 characters for rich text fields

Read the help docs on performance


NotesSQL Caveats

Problems with MS Excel 2003– Sometimes hangs while accessing or saving data– Access 2003 works fine though– Excel 2007+ also works fine

Problems with OpenOffice 2.x– View columns with text values are blank– OpenOffice 3 works, but has problems with very large views

Make sure users are using the correct views for lookups– View names are not always obvious in existing databases with lots of views

If users are sharing reports/spreadsheets, the NotesSQL DSN must be set up exactly the same for each user


From the Server: Domino JDBC

OpenNTF project written by Philippe Riand

No Notes client required!

Access Notes view data directly from the Domino server using JDBC


How It Works

Runs as a server task– Uses DOTS (another OpenNTF project) to allow Java to run as a server task– Domino 8.5.3 or higher

Define which views to expose as tables using an XML file in the database

Creates a SQLLite “virtual table” for each view

JDBC driver on the client reads the data from SQLLite tables on the server

NotesViews

SQLLite Tables

JDBC Driver


Setup: Install DOTS (Windows example)

Download from OpenNTF http://openntf.org/p/OSGI+Tasklet+Service+for+IBM+Lotus+Domino

Unzip to c:\dots

Copy these files from “c:\dots\build\win32" to the Domino program directory: dotsExtMgr.dll, dotsNSFHook.dll, ndots.exe

– Or build\win64, or build\linux64, or whatever

Create the following folders in the Domino program directory: osgi-dots/shared/eclipse/plugins and osgi-dots/rcp/eclipse/plugins

Copy these files from “c:\dots\build" to the Domino\osgi-dots directory: launcher.jar, com.ibm.notes.java.api_XX.jar, and dotssec.jar

Copy c:\build\com.ibm.dots_XX.jar to osgi-dots/shared/eclipse/plugins


Setup: Install DOTS

Download Eclipse 3.7.x (Indigo) from http://www.eclipse.org/downloads/packages/release/indigo/sr2

– Get the “Report Developers” version, because we’ll be using BIRT later

Unzip Eclipse– Use 7-Zip, not the native Windows unzip program

Copy required org.eclipse.* files from Eclipse to osgi-dots/rcp/eclipse/plugins– These MUST be from Eclipse 3.6.x or 3.7.x. Other versions of Eclipse might not work,

and the versions already in the osgi directory on the Domino server definitely will not work

org.eclipse.update.configuratororg.eclipse.osgiorg.eclipse.equinox.preferencesorg.eclipse.equinox.registryorg.eclipse.equinox.app

org.eclipse.equinox.commonorg.eclipse.core.runtime.compatibility.authorg.eclipse.core.jobsorg.eclipse.core.runtimeorg.eclipse.core.contenttype


Setup: Install DOTS

Add the following lines to notes.ini on the Domino server:– NSF_HOOKS=dotsNSFHook – EXTMGR_ADDINS=dotsExtMgr

Start the Domino server

Type "load dots" at the server console

You should get a "Domino OSGi Tasklet Container started" message

– If you copied the samples.jar file, you will also get lots of messages every 60 seconds

To stop the DOTS service, type "tell dots quit"


Setup: Install JDBC

Download from OpenNTF: http://openntf.org/p/JDBC+Access+for+IBM+Lotus+Domino

Unzip to c:\DomJDBC

Copy the following folders from {domino}/osgi/shared/eclipse/plugins to {domino}/osgi-dots/shared/eclipse/plugins :

– com.ibm.commons.jdbc_XX– com.ibm.commons_XX

Copy the following file from {domino}/osgi/rcp/eclipse/plugins to {domino}osgi-dots/shared/eclipse/plugins :

– com.ibm.notes.java.api_XX.jar– If you already have the java.api file from the DOTS install, just remove the existing one

and replace it with the one from Domino


Setup: Install JDBC

Copy all the plugins from c:\DomJDBC\updateSite-dots.zip to {domino}/osgi-dots/shared/eclipse/plugins

Copy com.ibm.xsp.extlib.relational.domsql_1.0.0.jar from c:\DomJDBC\updateSite-xpages.zip to {domino}/osgi-dots/shared/eclipse/plugins

Restart DOTS

You should get a "Initializing DomSQL RMI server for Domino" message


Setup: Install JDBC

The default JDBC port is 8089, and you need to make sure your firewall allows ndots.exe to use that port

– You can change the port number by adding DomSQL_Port=XXXX in notes.ini


Configuring Your Databases

Notes databases on the Domino server must be explicitly set up to allow JDBC access to views

– Each database set up individually– You can set up a single view, or all views, or a group of views

Security– Only expose the views you want to expose– Database ACL is respected

As with ODBC, you might want to create special “reporting” views



Open the database in Domino Designer

Add the "Package Explorer" view if you don't already have it (Window - Show Eclipse Views - Other - Java - Package Explorer)

Expand the database in Package Explorer and go to Web Content - WEB-INF



Create a new folder under WEB-INF called "jdbc“

– Right-click - New - Other - General - Folder

Create a new file in the new "jdbc" folder called "views.domsql“

– Right-click - New - Other - General – File– It doesn’t specifically need to be called

“views”, it can be anything with a .domsql extension

– You can have multiple .domsql files



To expose all views via JDBC, just use this in your .domsql file:

To expose specific views, rename views, override column properties:



NOTE: DomSQL uses the "Programmatic Use" name for columns in the view. You might want to override this either in the view itself or using a <column> definition in the .domsql file.


Example: Connect using SquirrelSQL

Many programs are available to access data via JDBC

One nice (free) one is SquirrelSQL– Java, LGPL– Plugins– SQL code completion– Export SQL query results to CSV, Excel,

XML format– http://squirrel-sql.sourceforge.net



Download and Install SquirrelSQL

Copy the com.ibm.domino.domsql.driver_XX.jar file to SquirrelSQL/lib

Start SquirrelSQL

Add a new driver with the following properties: – Name: Domino JDBC Driver– Example URL: jdbc:mysql://<hostname>[<:8089>]/<dbname.nsf>/<.domsqlname>[?

<user>=<value1>][&<password>=<value2>] – Class Name: com.ibm.domino.domsql.DomSQLDriver



Choose the menu option Aliases - New Alias. Create an alias with the following properties:

– Name: LS13 Test– Driver: Domino JDBC Driver– URL: jdbc:domsql://localhost/ls13/SHOW104DuckData.nsf/Views

(“Views” is the name of the .domsql file)



Choose the menu option Aliases – Connect



Use the Content, Row Count, etc. tabs to view information

– Note that column data types are automatically determined

Go to the SQL tab to run queries

Queries can be exported as CSV, Excel, or XML

– Use the “Store Result of SQL in File” toolbar button


Example: Create a BIRT Report

BIRT: Business Intelligence and Reporting Tools– http://www.eclipse.org/birt

Eclipse-based open-source reporting system– Originally developed by Actuate– Now a top-level Eclipse project

Can install stand-alone, as part of an Eclipse installation, or as a runtime for J2EE integration

Reports can be published to a reporting server, or exported as PDF, HTML, Excel, and other formats



Since we already downloaded Eclipse 3.7 for Report Developers (which includes BIRT) as part of the DOTS setup, we will use that

Launch Eclipse

Create a new workspace

Go to the workbench



File – New – Other – Business Intelligence and Reporting Tools – Report Project



Give the project a name

Click “Finish”

Switch to the Reporting Perspective when prompted



File – New – Report



Give the report a name

Choose the “Chart & Listing” template

Make sure “Show Report Creation Cheat Sheet” is checked



Create a new JDBC Data Source



Use “Manage Drivers” to use the Domino JDBC driver



Click “Add” to specify the location of the DomSQL driver



DomSQLDriver should now appear in the Driver Class list



Create a new Data Set using the Data Source



Create a SQL Query to define the Data Set



Right-click the table on the report layout view and choose "Edit Data Binding"



Select the data set you just created and click OK



You can now drag and drop fields from the data set onto the report and chart



Many formatting options for the report itself– Columns– Headers/Footers– Grouping– Page breaks– Charts

Many output options for the final report– PDF– MS Office (DOC, XLS, PPT)– OpenOffice (ODP, ODS, ODT)– HTML– Also send to a BIRT reporting server


JDBC Optimization and Tuning

SQLLite tables can be created as temporary, memory, or file

Sorted columns can be used as indexes

Use indexed fields for JOINs

Avoid using COUNT in your SQL queries (let the reporting tool do that work)

Use the Debug page in the example database for testing– And the NATIVE_TRACE_PERFORMANCE_HINTS option

You can create a .jdbc file if you want to add authentication info

Read the PDF that comes with the project for more tips


JDBC Caveats

Windows-only right now

You really should set this up on a separate reporting server, rather than a normal production server

– There can be memory issues– The RMI task doesn't shut down properly unless you shut down the entire server.

Using default views along with the <view> definitions will result in duplicate tables shown by the reporting client

If you have "Connection refused" errors, make sure DOTS is running!– You can load DOTS/JDBC when the server starts up by adding "dots" to the ServerTasks

line

No connection is info shown on the Domino console

If you change your .domsql files, you have to restart the Domino server


Contrast and Compare: NotesSQL vs. Domino JDBC

Notes SQL– Notes client required

– No server configuration required

– No database configuration required

– Server or local databases

– XPages integration NOT recommended

Domino JDBC– No Notes client required

– Server configuration required

– Minor database configuration required

– Server databases only

– XPages integration


Agenda

Overview

Working with Feeds







Issues Creating Files on the Domino Server

Sometimes you need to create a file directly on the Domino server– Dynamic file generation for web page– Create a file and attach it to an email– A present for your Domino Administrator

With agents, you need to have:– Unrestricted Access in the Security section of the server doc– Restricted operations allowed in Agent security

With XPages, you need to have:– Permissions set in the /jvm/lib/security/java.policy file

There are creative ways around this in certain situations…


Example #1: Streaming a File from an XPage

In the first example, we want to generate an Excel spreadsheet on-the-fly and send it directly to a web user

From a process perspective, it will be:– The user clicks a link that says “Download Spreadsheet”– The server creates a spreadsheet based on a view or custom parameters or whatever– The user gets the “Where would you like to save this file” prompt

There are ways to trick Excel into thinking an HTML table is actually a spreadsheet file

– See David Leedy’s http://xpagescheatsheet.com for an example

We want to generate an actual binary file and send it for this example


Apache POI for Writing Native Excel Spreadsheets

http://poi.apache.org

Java only, no DLLs or MS Office installations required

You only need to make a single JAR file (poi-XX.jar) available to your code

Creates an actual binary Excel file, not a text file that can be opened with Excel– Can read as well as write– Formats: XLS, XLSX, DOC, DOCX, PPT, PPTX– Also additional libraries for Visio, Publisher, and TNEF

Interesting examples in the HSSF (Office) and XSSF (Office XML) samples file– Multi-sheet calendar– Formatting, calculations, and freeze-panes– Convert DOC or XLS to HTML for web display


Attach the POI JAR File to the Notes Database

We will attach the Apache POI JAR file directly to the Notes database so it can be used by the XPage

Open the database with the Package Explorer view

– Like we did in the JDBC example

Navigate to the WebContent/WEB-INF folder

Right-click WEB-INF and choose New – Other…


Create a New Folder in WEB-INF called “lib”


Copy the POI JAR File to the New “lib” Folder

Right-click the new “lib” folder and choose the menu option “Import”


Find the POI JAR File and Attach It


Add the “lib” Folder as a Source Folder


Add the POI JAR File to the Build Path

This allows the Java file we will write to use the JAR for compilation


Add the POI JAR File to the Build Path


Create A Java Class

Next we need to create a Java class that can be used by an XPage

Java design element introduced in Notes 8.5.3

For older versions, you can create a class directly in the “lib” folder in Package Explorer


Create A Java Class

Needs a package name and a class name


Create A Java Class

Add code to generate an Excel file using POI


Create A Java Class

The code writes to an OutputStream provided by the caller (SSJS from an XPage)


Create an XAgent XPage

Create an XPage with a catchy name



We need to modify the source of the XPage to turn this into an XAgent– http://www.wissel.net/blog/d6plinks/shwl-7mgfbn

This allows us to set the HTTP response headers, get a handle to the XPage OutputStream, and call the Java code we just wrote


When Called from a Browser:


What Just Happened?

The XPage used HTTP headers that indicated that this was an Excel file, and it was a file download

The XPage called the Java class, passing it a reference to a Notes View and the XPage OutputStream

The Java code generated a spreadsheet in memory and wrote the bytes directly to the XPage OutputStream

The browser client received a file without ever having the file written to disc


Example #2: Generating and Emailing a PDF

In the second example, we will generate a PDF on-the-fly and send it as an email attachment

We will use an agent, so you have plenty of different ways to call it

Options for accessing a JAR file from an agent:– Attach directly to an agent– Attach to a script library– Add to the jvm/lib/ext folder on the server

The trick here will be attaching to an email (or any Notes document really) without creating a file on the server


Java PDF Libraries

Several to choose from:– Apache FOP– Apache PDFBox– PDFJet– iText– Many others…

Always check the license

Lowagie iText is an excellent option– Initial release in 1999– Currently on version 5.x (although it jumped from 2.1.7 to 5.0.0 in 2009)– Open-source licensed as AGPL in version 5.0, dual-licensed as MPL/LGPL for version

2.x– http://itextpdf.com


Options for Creating a New PDF in iText

Generate the PDF from scratch– Create a PDF doc in memory– Add formatting, paragraphs, images, etc.– Save to a file

Use an existing PDF as a template– Interesting technique described by Jake Howlett (codestore.net)– Create a template doc in Symphony or Word or whatever, and save to a PDF– Open the PDF template file in iText– Add paragraphs, images, etc. as above– Save to a file


Creating a Template

To generate this stunning template:– I created a document in OpenOffice– Added a headline and a graphic– Used the “Export as PDF” option


Creating the Java Agent

We need to create a Java agent and attach both the PDF template and the iText JAR file to it


Add the PDF Template to the Agent


Add the JAR File to the Agent


The Agent Should Look Like This


Agent Code (Generate the PDF)


Agent Code (Attach the File)

The PDF generation method returns a byte array. Now we need a way to write a byte array to a RichTextField.


Agent Code (Attach the File)


The Final Result

You can certainly make it look prettier than this

iText has a lot of options– Text styles– Tables– Images– Barcodes


Agenda

Overview

Working with Feeds







Understand the Requirements

The scenario: someone says “I need that data”

Questions you should ask:– Is this an actual requirement? (sometimes it's not)– What are you doing with the data?– How often?– What else will you need?

Understand sorting, grouping, categorization

Is it for reporting, app integration, or migration?

Don’t set yourself up for failure

Reliability Reliability Reliability


Understand the Users

Is this for a person, a department, or a committee?

Are you working with another developer, or a lesser being?

Does the user have a specific requirement or is this a fishing expedition?

Have them walk you through the entire story of what they need– Maybe they really need something different than what they're asking for– Users don't generally know what their options are

If the user fails, you have failed

Data feeds loose their WOW factor very quickly and become utilities just like electricity: under-appreciated but completely indispensable


How Much Code Should You Write?

As little as possible with out sacrificing reliability

Always consider the "no-code" option first– View export or copy-as-table– NotesSQL– ReadViewEntries (if it's another dev who needs it)– No-Code means there's less to break

How often do you think the requirements will change?– They will always change at least once– Frequent changes == spend time upfront writing flexible code

Are you dealing with rich text or attachments?

The older the data, the more messed up it will be


Try Not to Become a Report Writer

Try to make your code run without maintenance or constant updating

Try very hard to keep a separation between data and reports– You are in charge of data– Someone else is in charge of reports– This is even true for migrations

Business users probably know the data better than you anyway

Writing reports is a slippery slope– "Can I have another one sorta like that, but...?"– "Just like this, but weekly instead of monthly."– And just one more...– And just one more...

You gotta do what you gotta do, just watch out– Data is plenty of work all by itself


Normalizing and Scrubbing Data

Consider which fields could be categories– That data needs to be consistent

Are numbers and dates really numbers and dates?– You'll need to run pre-exports

Fix it in the database or in the export?


Use a Dedicated Report Server when Possible

Should it be replicated or copied data?

Make sure replicated data is one-way unless it specifically needs to be otherwise

Sometimes copies are better– No risk of deleting/changing production data– Re-run processes over and over and always expect the same data

Much less concern over slow processes, hangs, crashes

Admins more willing to "loosen" security– Agent permissions– Java security file– Installing external JARs, DLLs, etc.


Agenda

Overview

Working with Feeds







Links and References

Sample database and slides– http://londc.com/ldc.nsf/pages/goodies

Overriding Session Authorization on Domino: – http://publib.boulder.ibm.com/infocenter/domhelp/v8r0/topic/

com.ibm.help.domino.admin.doc/DOC/H_OVERRIDING_SESSION_AUTHENTICATION_8847_STEPS.html

Liquid XML Studio– http://www.liquid-technologies.com/xml-studio.aspx

Soap UI– http://www.soapui.org

Java Atom/RSS Feed Parsing: – ROME – http://rometools.org – Apache Abdera – http://abdera.apache.org – Apache Wink – http://incubator.apache.org/wink


Links and References

XAgents– http://www.wissel.net/blog/d6plinks/shwl-7mgfbn – http://www.mindoo.com/web/blog.nsf/dx/17.07.2011101855KLEBRW.htm

NotesSQL– http://www.ibm.com/developerworks/lotus/products/notesdomino/notessql

Domino JDBC– http://openntf.org/p/JDBC+Access+for+IBM+Lotus+Domino

SquirrelSQL– http://squirrel-sql.sourceforge.net

BIRT– http://www.eclipse.org/birt

Apache POI– http://poi.apache.org

iText (Java PDF generation)– http://itextpdf.com


Related Sessions

AD206 : IBM Lotus Domino XPages: Embrace, Extend, Integrate– Niklas Heidloff, IBM; Padraic Edwards, IBM

AD208 : IBM Lotus Domino XPages Performance in a Nutshell– Maire Kehoe, IBM; Tony McGuckin, IBM

AD204 : How To Develop Great Applications Using XPages Design Patterns– Tony McGuckin, IBM; Martin Donnelly, IBM

AD202 : Debug Server Side Javascript, Java, and XPages Apps Using the SSJS Debugger– Dan O'Connor, IBM; Michael Blout, IBM

ID110 : Deep Dive into IBM Lotus Notes Calendaring and Scheduling and Related Systems– Bruce Kahn, IBM

AD210 : It's Here! Calendar APIs And REST Services– Dave Delay, IBM; Nathan Barry, IBM

BP208 : XPages Blast– Tim Clark, TC Soft Consulting Limited; Matt White, Fynn Consulting Ltd

172 © 2013 IBM Corporation

Legal disclaimer• © IBM Corporation 2013. All Rights Reserved.

• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

• Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

• Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

• Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

• Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

• Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

• UNIX is a registered trademark of The Open Group in the United States and other countries.

• Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.

• All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.


THANK YOU!

Mark MyersLondon Developer [email protected]: @stickfight

Julian [email protected]: @jrobichaux

show104 - buried treasure: finding the hidden gold in lotus notes data

Technology