I-SPY – CS490BN ADVANCED JAVA TOPICS Derek Adams and Camilo Rostoker

I-SPY Final Report

Table of Contents

C H A P T E R 1

Executive Summary 1

C H A P T E R 2

Introduction 3

Purpose 3

System Overview 4

C H A P T E R 3

Server-Side Enterprise Architecture 5

Server-Side Libraries 6

Server-Side Components 8

C H A P T E R 4

Client-Side Architecture 15

Client-Side Libraries 15

Client-Side Components 16

Client Application 24

Web-based Client Application 28

C H A P T E R 5

Technologies 31

C H A P T E R 6

Future Enhancements 34

A P P E N D I X A

Testing 36

A P P E N D I X B

Class Diagrams

1

Executive Summary I-SPY is a multi-faceted Internet surveillance and monitoring system. The main purpose of I-SPY is to capture images from a capture device and send them across a network to a central server. The I-SPY design architecture was chosen to addresses two main issues: 1. Remote Access: Traditional image capturing software does not provide

and easy method to view the captured images from a remote location. Most home users do not run web-servers, and other video conferencing applications (e.g. NetMeeting) are not designed to be left running continuously in order to provide a means of remotely viewing the captured images.

2. Local Storage: Captured images are stored locally. This means that the

captured images can be easily erased from existence if the computer is stolen, destroyed, or the files are deleted manually.

The main purpose of this project was to develop a system that displays the usefulness of a distributed video surveillance system. By defining I-SPY as a distributed application means that its’ components reside on more than one computer system. The current configuration of I-SPY uses two locations: one for the client and one for the server. However, the server-side infrastructure was developed on a Java 2 Enterprise Edition (J2EE) infrastructure, and one its’ key features is that components can be built once and distributed among numerous machines with little effort. This means that in actuality, although I-SPY is currently only a two-tiered system, it has the capabilities to be distributed among n-tiers, giving it many of the highly desirable properties of a distributed system such as scalability, reliability and flexibility. J2EE is the rivalling platform to Microsoft's .NET. While this project was not concerned with showcasing any particular product, we include a small result from another paper that performed a detailed comparison of the two platforms:

Chapter

1

2

"The issue of portability is a clear victory for J2EE since it is based on the Java language, whereas .NET applications are limited to Microsoft platforms. Both J2EE and .NET contain facilities for integrating existing legacy host systems, however many people believe the facilities provided by J2EE are superior to those provided by .NET. Although J2EE masters portability, it consequently lacks support for programming languages other than Java, while .NET provides a run-time environment capable of executing code from a number of popular programming languages. J2EE has been used in production environments for several years now, including many large-scale projects from Fortune 500 companies. This extensive usage implies that many of the bugs and issues have long since been resolved, whereas .NET is new and untested, and will most certainly give rise to the usual heap of patches and updates one can expect from Microsoft products."1

The complete I-SPY architecture has three main components: Client Application, Server-Side components, and the client web-access to server-side services. The I-SPY client application resides on the same machine that where the capture device is installed. The main responsibility of the I-SPY client is to capture images and send them across the network. The client application provides a rich graphical user-interface that allows users to control the settings and behaviour of their webcam. The client is also equipped with remote management facilities for modifying local client settings and controlling the capture device from a remote location. If the client can't connect to the network, it can operate in Offline mode, meaning images are captured and stored locally for a user-specified amount of time. The two main server-side components are the Image Transfer Server (ITS) and the Enterprise Java Bean (EJB) container. The EJB container provides a robust and reliable environment for the two main EJB components that encapsulate the I-SPY business logic. EJB's are sophisticated, re-usable components offering many useful advantages for enterprise-level systems. The I-SPY client web-tier is a web application that also interface's with the I-SPY server-side components. I-SPY’s web-based client provides user’s with the ability to view the incoming images, maintain an image archive, as well as remotely control their capture device.

1 “J2EE vs .NET - The Final Showdown”. Rostoker C., University of Technology (Sydney), 2002.

3

From concept to implementation, this project has seen many glorious moments, and many challenges encountered. While the current system exhibits many positive aspects and powerful features, there is inevitably more work to be done before the system can be considered complete. Please refer to the Future Enhancements section near the end for details on how the system can be improved.

4

Introduction I-SPY, the PC-based video monitoring or surveillance system

-SPY is a purely Java-based surveillance monitoring system. Goals of top priority include web integration using Java Server Pages. There are various major issues in developing this system, such as technology integration, design, process,

library availability, and infrastructure implementation.

Purpose As stated in the project proposal: “The main purpose of this project is to develop a prototype system that displays the usefulness of a distributed video surveillance system.” The system resulting from the project is nearly fully functional and exhibits many properties of a distributed system. We therefore believe the project has successfully reached the proposed goals. This document contains information describing the technical details of the I-SPY system.

Advanced Java One of the primary goals of this course was to learn and utilize various

advanced Java libraries. Java technologies are perhaps the hottest technology at the moment in the I.T. industry, and therefore this project is a highly valuable learning experience.

The finished project integrates numerous advanced Java technologies to produce a distributed, multi-threaded, enterprise-level system. The main components on the server side utilize the J2EE infrastructure, while the client side application takes advantage of the powerful Java Media Framework (JMF). The overall system uses a combination of several communication protocols, including TCP/IP, RMI and HTTP.

Chapter

2

I

5

I-Spy Architectural Overview

The diagram below illustrates a high-level design overview of the complete I-SPY architecture, depicting the main components and protocols used throughout the system.

6

7

Server Side Architecture I-SPY is a J2EE-based System

Server-Side Enterprise Architecture n order to setup the server side infrastructure, suitable hardware was required. The solution was simple – purchase a PC. The OS that was installed was Slackware

LINUX, version 8.1. The JDK installed was the latest from Sun Microsystems. The new machine was named Akuma and resides on the helius.ca domain.

Enterprise Infrastructure Part of the infrastructure setup including installing and configuring J2EE. This package includes all necessary tools and libraries required to run any J2EE tool, including Application servers such as Tomcat. And Jboss. The version of J2EE that is running on Akuma is version 1.4.

Apache Tomcat is a pure Java-based application server. Tomcat “serves” JSPs and Servlets. Tomcat can be integrated with Apache, but currently on Akuma it is responsible for both HTML and JSPs. For practice, www.helius.ca was rebuilt using JSPs.

JBoss is an Open Source Application Server. That is, JBoss is the container for EJBs (Enterprise Java Beans). An EJB container is the component that provides a runtime environment for a particular

Chapter

3

Akuma Specs: Pentium II 450 Ram 256MB OS: Slackware 8.1 JRE: Sun JDK 1.4.1_01

I

8

EJB. Also provided with JBoss is the ability to integrate with Tomcat, allowing I-SPY to utilize both JSPs and EJBs, thereby covering the core components of J2EE.

One other important tool that is not part of J2EE or Tomcat, but part of a quality Java Development environment, is Apache Ant. This tool provides a standardized building process for the I-SPY server-side software. Ant is used to both build the software and generate the Javadocs API.

Server-Side Libraries J2EE can contain a variety of components, each one of which services a different layer as illustrated below by FIGURE 1. The I-SPY architecture includes a component at each of the J2EE layers.

§ On the client side, I-SPY utilizes J2EE both on the user Desktop and through a web browser.

§ On the Server-Side Presentation Layer, JSPs are used to communicate with standard Java2 libraries as well as Enterprise Java Beans (EJBs) and the client on the web browser.

§ On the Server-Side Business Logic Layer, EJBs are used to encapsulate the I-SPY business logic.

FIGURE 1 A JSP is compiled into a Java Servlet by the Application Server. For this reason I-SPY’s web tier could have been entirely developed in Java Server Pages. Diagrams provided by Sun Microsystems2.

2 J2EE Overview: http://java.sun.com/j2ee/overview2.html

9

Web Tier Development of the web tier, that is the presentation of I-SPY on the web, is accomplished through Java Server Pages. Java Server Pages is a technology that allows a developer to embed Java code directly inside HTML or XML. Traditionally, the web was a network of static pages, with little-to-no dynamic content. Common Gateway Interface (CGI) was an early attempt at a solution to this problem, where developers wrote individual programs to interface with the web. However, each CGI request launches a new process on the server, and results in poor performance. In addition, there are numerous reliability and security issues with CGI’s. Java Servlets solve some of the problems inherent in CGI. However, a Servlet is similar to a traditional CGI script, in that when it is run, it generates mark-up text that is to be read by a browser. With this approach, the entire page must be composed in the Java Servlet. If a developer wants to change the page, they would have to edit and recompile the Java Servlet, even if the logic is already working.

Java Server Pages have the following advantages:

§ Separates content generation from presentation

Developers only need to be concerned with the presentation of the page, as they understand that the application server will encapsulate the content generation. Also, the use of Beans in JSPs further encapsulate the logic of the application

§ Reusable Components

Developers can deploy reusable Enterprise Java Beans to perform the more complex processing of the application.

§ Simplify Page Development

Java Server Pages encapsulate much of the dynamic code generation in JSP specific XML tags.

Business Tier I-SPY’s Business Tier, that is the logic tier of the server side components, will be a combination of JSPs, standard Java2, and Enterprise Java Beans (EJBs). Enterprise Java Beans are server-side components that encapsulate the business logic of an application.

10

According to Sun Microsystems3, you should consider using enterprise beans if your application has any of the following requirements:

1. The application must be scalable. To accommodate a growing number of users, you may need to distribute an application's components across multiple machines. Not only can the enterprise beans of an application run on different machines, but their location will remain transparent to the clients.

2. Transactions are required to ensure data integrity. Enterprise beans support transactions, the mechanisms that manage the concurrent access of shared objects.

3. The application will have a variety of clients. With just a few lines of code, remote clients can easily locate enterprise beans. These clients can be thin, various, and numerous.

Although I-SPY at the moment does not require good scalability, if I-SPY were to become a commercial product, scalability would definitely be important. At this moment, transaction support is not required.

EJBs will be used along side standard Java2 libraries on the server-side when needed. For instance, Entity Beans can be used as a persistence storage mechanism for web-based modules.

FIGURE 3 EJBs are embedded in JSPs for server-side logic.

Server-Side Components The I-SPY Server-Side Architecture takes advantage of the numerous facilities provided by J2EE. J2EE has a number of complex tools available to a developer, and is quite complex to set up. There is no single vendor that provides all the necessary tools. If J2EE developers take the “Open Source path”, support is very poor, and that includes poor documentation

3 J2EE Tutorial: http://java.sun.com/j2ee/tutorial/index.html

11

of many of these tools. I-SPY completely takes the “Open Source path”, in hopes that in the long run, or in any future developments, the steeper learning curve will pay off. Why Use Enterprise Java Beans? Typically in the past, companies have written proprietary middleware. EJBs allow an organization to purchase off the shelf components from one vendor, and combine it with the components of another vendor. Furthermore, components can run in the application server of yet another vendor. EJBs are extremely portable, and incorporate a wide variety of concepts from many different areas, including distributed computing, databases (Entity Java Beans), security, and component driven software.

The goal of EJBs is to provide a convenient environment to create and deploy scalable, reliable, and secure applications, without the need of writing your own complex distributed component framework. With EJBs you can rapidly construct a server-side infrastructure by leveraging a prewritten distributed infrastructure

Note that after using EJBs, the authors would not recommend using them, unless absolutely required. They are difficult to write, debug, and deploy. There are other technologies that could easily replace the functionality of EJBs, such as Java Spaces.

12

Why Use Enterprise Java Beans in I-SPY? Perhaps the J2EE tool that is most complex is the Enterprise Java Bean. As do all J2EE projects using EJBs, I-SPY uses them on the server to encapsulate business logic. That being said, I-SPY could have been written without using EJBs, and in a much quicker amount of time. However, the goal was to learn EJBs - not necessarily to create I-SPY in the quickest amount of time. Most of the time spent on EJBs was simply learning how they work. Even assembling the most simple EJB took a vast amount of time.

I-SPY Infrastructure Set Up Setting up a J2EE environment is an involved process, even more so when one aspires to building an Enterprise Java Beans. Here is a summary of the process that was used to build EJBs for I-SPY.

? �To set up the J2EE infrastructure for I-SPY the following steps are required:

1) Install J2EE

§ J2EE must be installed in any J2EE environment, so that it’s numerous libraries are available to the developer and the J2EE applications.

2) Install a JSP Server

§ A JSP server is required for JSPs and Servlets, and therefore Tomcat was installed. In this step the developer must learn how to use Tomcat and how to write JSPs and Servlets.

3) Install a Enterprise Java Beans Container

§ To use Enterprise Java Beans (EJBs), which have a terribly steep learning curve, an EJB container is required. Chosen was perhaps the most difficult one to learn, JBoss. However the long term benefits outweigh the learning curve, as no purchase of software is required if commercial software development occurs for profit.

Deploying Enterprise Java Beans EJBs are perhaps one of the most complicated pieces of software around so the learning curve is long. To the uninformed, while it may appear that a developer new to Enterprise Java Beans has made little progress or little effort, this is certainly not the case.

? To deploy an EJB the following is required:

1) Read up, inadequate research, and lack of an understanding of the concepts of EJBs, will only result in failure. The Server Side has an excellent book regarding EJBs, which is free with registration.

13

2) Decide on the design. Design is crucial, as there are many different types of beans including (Stateful/Stateless) Session Beans, Transaction Beans, Entity Beans.

3) Write the Home interface. This is sort of like a constructor to a normal class, however this is kept in a separate file to that of the Enterprise Bean.

4) Write the Enterprise Java Bean itself. The code must follow the concepts for the type of bean being deployed.

5) Write the Remote Interface. To anyone who has used RMI, this is similar. Any methods that the developer wants to publish must be listed in here.

6) Write a local interface. This step is optional. I-SPY has no Local interfaces.

7) Write the deployment descriptor. This involves writing an ejb-jar.xml file.

8) Write the Container specific XML files.

9) Put the ejb-jar.xml file in a folder called META-INF in the directory of the jar that is to be built.

10) Next a build tool such as Ant is required, as building a bean, is not as simple as compiling just any file. With Ant script the build.

11) Finally once the JAR is finally built, deploy it, and this is simple, just copy it into the JBoss deploy directory.

12) Using the bean is not as simple as one might think. To use the Bean, requires the right imports, the right way to call the bean, and finally must include all the correct jars in the path. There are also JNDI naming issues.

Writing the EJBs ? Enterprise Java Beans typically Consist of 3 files:

1) The Enterprise Java Bean Object (EJB Objects): § The Enterprise Bean Object contains all code logic. That is the

actual class methods are implemented in this class. 2) The Home Interface: § A good way to think of this class is as a Factory of EJB Objects.

Via the Home Interface, EJBs Objects are created, EJB Objects are found, and EJB Objects are removed.

3) The Remote Interface: § Clients obtain references to EJB Objects through the Remote

Interface. Developers familiar with Java RMI, can see that this is similar to the “Service” Interface extending the java.rmi.Remote

14

interface. All methods available to the client must match the signature of the EJB Object. These are the methods that are available to the client to invoke.

15

AuthBean This bean coordinates access to a majority of the I-SPY business logic. In greater detail, it is responsible for maintaining and coordinating access to session profile information, user profile information, as well as providing security and creating Database manager objects. ITPServer creates a reference to an Auth bean. With this Auth reference, ITPServer signs into the I-SPY system, giving the Auth object it’s user’s profile information. Via the Auth bean, ITPServer can then obtain a DBManager object. ITPServer is also responsible for updating the information in AuthBean, such as with the latest file name so that the incoming images can be viewed from the web-client access.

Class Name Type Of Class Auth The Remote Interface AuthHome The Home Interface AuthBean The EJB Object

WebClient Webclient provides all the necessary services required of the JSPs to communicate with the server-side components. This includes login capabilities and communication with the AuthBean. Through the webclient, JSPs can view the directory listings of the database, view the size of the directories, view/update the user profile, obtain the session profile, etc.

Class Name Type Of Class WebClient The Remote Interface WebClientHome The Home Interface WebClientBean The EJB Object

Standard Java2 Classes used by I-SPY EJBs There are two fundamental classes that are fundamental to I-SPY

§ UserDBManager: Accesses a database that stores user information. This database is a simple XML file. Through this class XML tag values can be retrieved, created and updated. One interesting method to note is shown below:

public void setUserDBSetting(String username, String key, String value)

This method updates a value in the XML file. This could be a powerful

tool, as information, such as the last logon date, could be stored via this method.

16

§ DBManager: Creates the file structure for I-SPY database. That is, it creates the appropriate directories for each user where each file will be created. In addition files have a naming structure determined by DBManager.

17

Standard Java2 Classes used by I-SPY EJBs The web tier in I-SPY is developed mainly through the use of Java Server Pages (JSPs). By using Java Server Pages, web developers can provide dynamic content. Because JSPs are Java, they can reference Java objects including Enterprise Java Beans. The following is a brief bit of code demonstrating a Java Server Page: <%@ page language="java" %> <%@ page import="java.util.*" %> <jsp:useBean id="webclient" scope="session" class="I-SPYbean.webclient.WebClientBean"/> <% webclient.signIn(request.getParameter("username"), request.getParameter("password"),request.getRemoteHost()); if(webclient.isLoginSuccessful()){ session.setAttribute("username", request.getParameter("username")); %> <jsp:include page="main.jsp"/> <% }else{ %> <jsp:include page="login.htm"/> <% } %>

The first tag simply explains that is a Java Server Page, and any tag starting with ‘<%’ is a scriptlet, java code embedded in html. The second line is how imports work in JSPs. <%@ page language="java" %> <%@ page import="java.util.*" %>

The remaining lines in the code sample show how a Java Bean is accessed from a JSP. The object is used to sign in and to see if the sign in is successful. If it is include the main.jsp or else go back to login.htm . <jsp:useBean id="webclient" scope="session" class="I-SPYbean.webclient.WebClientBean"/> <% webclient.signIn(request.getParameter("username"), request.getParameter("password"),request.getRemoteHost()); if(webclient.isLoginSuccessful()){ session.setAttribute("username", request.getParameter("username")); %> %> <jsp:include page="main.jsp"/> <% }else{ %> <jsp:include page="login.htm"/> <% } %>

18

19

Client Side Architecture I-SPY is a JMF enabled Application

Client-Side Libraries

I-Spy and the Java Media Framework The I-SPY client takes advantage of the powerful Java Media Framework (JMF) for robust OO-access to a systems underlying media framework. JMF encompasses video and audio, and can be used in a wide variety of applications. The diagram below illustrates the general JMF architecture:

FIGURE 4 Java Media Framework

Although this report will not discuss the technical details of JMF, please refer to the JMF homepage4 for excellent documentation, API’s and tutorials.

4 JMF Homepage: http://java.sun.com/products/java-media/jmf/

Chapter

4

20

Capture Device State There are a variety of states that the capture device must go through before it is ready to be used. In particular, I-SPY takes advantage of JMF’s concept of a Processor, which is a component used to read the raw video stream and perform various user-defined processing. A processor is a type of Player, which is the core concept used to read and render the raw video stream. The difference is that a Processor allows the user to control the processing and rendering methods, whereas the Player does not. Below is a state diagram of the Processor module:

FIGURE 5

In order to synchronize these states with the I-SPY system, Java’s built in Thread synchronization methods such as wait() and notify() are used to communicate between Threads. (Note: the reason the Configuring and Configured states are highlighted is because these two states are only valid states of the Processor)

Client-Side Components

The I-SPY client side consists of a highly integrated architecture of components, each of which serves a specific function. The overall process of the client side involves capturing frames (image) from a video stream, transmitting each frame to a remote location, configuration of configuration settings, and monitoring and maintenance of the local image cache. Each of the core components is discussed below.

The Client Core Each individual client-side component is contained within the ClientCore. The ClientCore serves as a “container” class that encapsulates and coordinates the various components of the system. Although each module serves a specific function, they are all derived from the base class

21

CoreModule, which serves as a common interface for all the core client modules. Each module is discussed in detail below.

Client Core Modules

Image Buffer The image buffer is a data structure used to represent a buffer with First-In First-Out (FIFO) queue sequencing. This component is used by both the Webcam Manager and the Image Transfer Agent. The Webcam Manager “produces” images, while the Image Transfer Agent “consumes” images and transfers them across the network.

Below is an illustration of a typical consumer-producer relationship:

FIGURE 6 Consumer-Producer diagram5

Synchronization Control To abstract the notion of the producer-consumer, the above synchronization mechanism is encapsulated within the ImageBuffer class itself, which releases the WebcamManager and ImageTransferAgent components from having to deal with synchronization issues.

5 Producer-Consumer diagram borrowed from: http://www.cs.mtu.edu/~shene/NSF-3/e-Book/SEMA/TM-example-buffer.html

22

There are two main methods within the ImageBuffer class that are used to provide the concurrency control needed to regular access to the images in the buffer; they are addImage and getNextImage. The WebcamManager class calls the addImage method to add in image into the buffer. If the buffer is found to be full it will call the wait() method which tells the invoking thread to wait and do nothing until notified. When it is notified and the buffer is not full, it will add an item to the buffer and then send a notification signalling that an image has been added to the buffer.

public synchronized void addImage(File image_file) throws ImageBufferException { while (isFull()) { printLocal("Full buffer...waiting for it to become non-full"); try { wait(); } catch (InterruptedException ie) { // an error occurred while waiting return; } } // now add to the end of the list items.addLast(image_file); // send a notify event to tell sleeping threads there is // now an image to consume notifyAll(); }

The Image Transfer Agent calls the getNextImage to retrieve a reference to the next image that is to be transferred across the network. This method works in a similar fashion to addImage except that it waits patiently until there is an item in the buffer to retrieve. Once it has taken an item from the buffer, it sends a notification signalling that there is one less item in the buffer.

23

public synchronized File getNextImage() throws ImageBufferException { // if there's nothing in the list, just chill while (isEmpty()) { printLocal("Empty buffer...waiting for producer..."); try { wait(); } catch (InterruptedException ie) { // an error occurred while waiting return null; } } // remove and return from beginning of list File f = (File) items.removeFirst(); // send a notify to waiting threads notifyAll(); return f; }

Image Buffer and Network Mode As mentioned before, I-SPY can be in one of two network modes: online and offline. The Image Buffer works slightly different depending on the selected network mode. In fact, the isFull method is the only differentiating method. When in online mode, isFull checks to see how many images are in the buffer and returns true (i.e. the buffer is full) if buffer_items >= MAX_ITEMS. However, when in offline mode, the isFull method will always return false, because the image buffer acts as a circular queue. If the buffer is full and offline, then isFull will first delete as many items as needed (oldest first) until there is space in the buffer.

Webcam Manager This class is responsible for the operation of the capture device. In particular, this component is responsible for capturing single frame images from the video stream and converting them into java.awt.Image objects. WebcamManager States The WebcamManager has its own set of states as follows: State Description STATE_OFF Capture device is not initialized and is disconnected from the video

source

STATE_INIT The capture device has been initialized using settings from client config but is not connected to the video source

STATE_ACTIVE The capture device is connected to the video source and is currently

24

capturing images

STATE_PAUSED Not capturing images but connected to video source Controlling the Capture Device The methods below are provided by the WebcamManager class to control the capture device:

Method Description Pre-State Post-State

startCapture Starts the capture process: captures frames as images, adds them to the ImageBuffer, and then using a supplied ImageWriter object actually writes the image into an image file on disk.

STATE_INIT STATE_ACTIVE

stopCapture This stops the capture process and disconnects the video source.

STATE_ACTIVE, STATE_PAUSED

STATE_OFF

pauseCapture Tells the WebcamManager to pause from capturing images, but does not disconnect the video source

STATE_ACTIVE STATE_PAUSED

restartCapture Tells the WebcamManager to stop capturing, disconnect the video source, reload configuration, re-initialize the capture device, and finally restart capturing images.

STATE_ACTIVE, STATE_PAUSED, STATE_OFF

STATE_ACTIVE

Capturing Images Images are created by capturing a single frame from the video stream. There are various options that are specified prior to initializing the capture device that dictate the resulting image attributes. For this project, the user is restricted to specifying a subset of these options. Option Description Valid Values Frame Rate Specifies how often to capture a still image from

the webcam and add it to the ImageBuffer. This value will have a realistic minimum value (e.g. one snapshot every few seconds) in order to satisfy the synchronization issues of producing images versus consuming images and transferring across the network

Min = 1/10 second Max = 1 hour

Frame Size This option defines the actual pixel dimensions of the frame. Frame sizes will be controlled, with a set of options to choose from.

640x480 320x240 160x120

25

Color Depth

This specifies the color-depth of the image. Color depth means how many bytes are used to store the color value at each pixel.

16 bit 24 bit 32 bit

Space Requirements – Image Size and Quality vs. File Size The frame size and colour depth correspond directly to the final size of the image file. The frame size depicts how many pixels are used to represent the raster image, while the colour depth results in higher quality images. At this stage, these values define how the raw image data is captured from the camera. Fortunately, the feature enabling acceptable performance over the network is the use of advanced imaging technologies to further compress the image into smaller and well-known formats. However, due to the nature of this project, a happy medium must be found that enables users to produce acceptable quality images at a decent rate, but still be able to transfer them over the network at relatively decent speeds. These two options, along with the frame rate, must be properly coordinated to ensure best performance.

Image Transfer Agent The Image Transfer Agent is responsible for transferring the captured images across the network to a remote destination. By using a reference to the ImageBuffer, the Image Transfer Agent is able to “consume” images from the buffer as they become available.

The Image Transfer Agent follows a strict protocol as outline by the Image Transfer Protocol. This includes establishing a TCP socket connection with the remote host, authorizing the user with a username and password, and then sending the images one at a time until either side terminates the connection. Details of the protocol are discussed in the Image Transfer Protocol section. The Image Transfer Agent can be in one of three states. The table below summarizes each state: State Description OFF The agent is initialized but is not doing anything (i.e. the Thread is not in

the run() method)

ACTIVE The agent is actively in the process of transferring images. To be more specific, the agent could be in the action of actually writing the image data through the network or it could be waiting for the next available image to become available. (This may need to be changed later on to be able to distinguish between these two states)

PAUSED The agent can enter the PAUSED state if it is the ACTIVE state, where it will sleep for a specified amount of time before checking its state again to see if it has become ACTIVE so that can continue transferring images.

26

User Profiles Currently, I-SPY uses a Java implementation of a Hashtable (java.util.Hashtable) to store the user profiles. The Hashtable provides the basic structure needed, which is to store various objects/values for each user that can be retrieved by specifying the key (e.g. to retrieve the user’s password a simple Hashtable method such as myProfileHash.get(“password”) will retrieve the corresponding users password). This decision also allows extensibility as the user profile attributes can be modified without having to explicitly change the user profile – each new attribute is just added as a new entry in the user profile Hashtable. Client Config This class is mainly used as a container class to store the various client options. The options are loaded and parsed from an XML file. Using the same design principle as the User Profiles (discussed above), a Hashtable is used as the underlying data structure to hold the configuration keys and values. Each core module receives a reference to this object so that can have access to any configuration information it may need. By passing it as a reference, the object can be modified/reloaded by any module and changes will be reflected in all other modules. Below is a portion of the config.xml file used to store the client configuration:

<?xml version="1.0" encoding="UTF-8"?> <config> <network_mode>online</network_mode> <itp_host>24.72.11.77</itp_host> <itp_port>9006</itp_port> <username>camilo</username> <password>12345</password> <max_image_buffer_size>10</max_image_buffer_size> <frame_interval>5000</frame_interval> <jpg_quality>80</jpg_quality> <color_depth>16</color_depth> <resolution_y>240</resolution_y> <resolution_x>320</resolution_x> <image_dir>c:\I-SPY\images</image_dir> </config>

ImageWriter I-SPY encodes each image as it is captured. The WebcamManager module contains an ImageWriter reference, which is an interface for all encoding type classes. The interface is quite simple, specifying only a single method that must be implemented. This OO design decision allows for flexibility in adding additional modules for various encoding/compression formats. The actual code for the interface is shown below:

public abstract interface ImageWriter { public void writeImage(Image img, String dest_filepath);

27

}

JPEGWriter The JPEGWriter class implements the ImageWriter class by defining its own writeImage method, in which an Image object is taken and converted to JPEG format and saved to disk. This is component in I-SPY that utilizes the Java Advanced Imaging Library (JAI) in order to manipulate the image as desired. Although writing image files is quite complex, I-SPY currently takes a simple approach by limiting a user’s input on the encoding parameters to just the Quality (other image attributes are specified in the WebcamManager module such as frame size & color depth) A more detailed investigation of the image quality will be conducted for the final revision of this document, which will compare various quality settings, file size, and network transfer results in order to obtain maximum quality while satisfying minimum bandwidth requirements.

Remote Client Management Services The I-SPY client is equipped with remote management services. These remote services are offered by using Remote Method Invocation (RMI) to encapsulate the client services within remote objects. The remote objects are then made available to remote clients so that local client services can be invoked from any location.

A ControlServer component is run in parallel to the client application. This component starts a RMI registry, creates the remote objects, and then publishes them to the registry.

There are currently two remote objects used to provide the remote client services: ConfigControlService and CoreControlService. The ConfigControlService object provides remote methods for retrieving and updating the client configuration. The CoreControlService object provides remote methods for core services, such as starting, stopping and pausing the capture device. Below is an outline of the remote methods available for each object:

Remote Methods of ConfigControlService public boolean pushConfig(Hashtable hashtable) throws RemoteException; public Hashtable pullConfig() throws RemoteException;

Remote Methods of CoreControlService public boolean startCam() throws RemoteException; public boolean stopCam() throws RemoteException; public boolean shutdown() throws RemoteException; public boolean pause() throws RemoteException;

28

29

The Client Application

The client application is a fully featured and functional Java application. The GUI is constructed using Java Swing, a platform-independent windowing toolkit. As depicted in the screenshot above, the GUI has a main toolbar on the top, a middle section, and a bottom component containing the image buffer. The middle components seen in this diagram are only part of the Console module and are replaced with new content when one of the toolbar buttons is clicked. There are four main modules within the application: Console, Local Image Cache, Settings, and Help. Each of the modules is discussed below.

30

Console Live Video tab Snapshot tab

The Live Video tab uses a MediaPlayer bean to render and display the video source. The MediaPlayer bean is a pre-assembled component (see section on Beans) that encapsulates the media control functionality.

The Snapshot tab paints the most recently captured image to the panel and embeds a time and date watermark so that the user can determine when the snapshot was taken.

Local Image Cache The Local Image Cache gives the user details about the where and how many images are stored. There are four fields in this screen:

• Cache Location: Where the images are being stored on local disk (specified in the config.xml file)

• Old Cache Size: How many images were already in the cache was when the client was started.

31

• Session Size: How many images are being stored locally from only the current session

• Total Cache Size: The total number of images in the cache (Old cache size + session size)

On the right side is a display that shows scaled-down versions of the session images (i.e. images as indicated by the Session Size). The image display can be refreshed by clicking on the ‘Refresh Image Display’ button.

Settings

The settings panel provides a mechanism for the users to specify various settings and options related to the function and behaviour of the client application.

The values are pulled from the ClientConfig object (which pulls from the config.xml file) and are loaded into the corresponding fields. When the ‘Apply Changes’ button is clicked, the values are stored back into the ClientConfig object which then writes the settings back to disk.

A description of each field is given below.

• Network Mode: Specify if the application should run in Online or Offline mode.

• Capture Device: Displays all capture devices detected on the system.

• Image Dimensions: Specify the dimension of the video/image produced by the capture device.

• Capture Interval: Specify how often an image should be captured. • Color Depth : Specify the number of bits per pixel in the

video/images • Compression Format: Specify what compression format should be

used to compress the images before storing locally or sending

32

across the network (Current version of I-SPY only allows JPEG compression)

• Image Quality : Specify the JPEG compression image quality (applicable depending on the Compression Format)

• Username: The username needed to connect to the Image Transfer Server

• Password : Password needed with the username Help The help panel is simply a convenient method for displaying help information. The help panel is created from a TextPane object. A TextPane object encapsulates a tradition Document Object Model (DOM) object that can represent a HTML document. Thus the entire Help content is just an HTML file that is loaded into the display area. The advantage of doing this is that the contents of the Help file can be easily modified and updated without requiring any changes to the client application. Each time the Help panel is displayed the corresponding HTML Help file is loaded from disk.

33

34

Client Web-Access

I-SPY provides users with web-based access to the server-side facilities. This means users can access their account from anywhere at any time provided they have Internet access and a web browser. Similar to the client application, the web client has three main modules: Console, Settings and Archives.

Console

The web console provides stats on their account, current stats on the active client session, remote device control, and viewing of captured images as they are transferred from the client application to the server.

The console page behaves differently depending on if the client is connected or not. Below is a screenshot of the console when the client is not connected.

If the client application is not connected, then the web client can’t communicate with the client application and the controls are unavailable. The same applies for the incoming image view – if the client is not connected then obviously there are no incoming images.

However, if the client is connected, then the capture device controls can call a remote method on the client using RMI, to either start, stop, pause or restart

35

the camera (currently the Stop command is disabled because it disconnects the client from the server thus rendering the remote administration useless).

Similarly, if the client is connected its’ configuration settings can be retrieved via RMI and displayed on the web. The user is then able to make changes and then save them back to the remote client. Obviously some configuration options should not be changed remotely, therefore only certain values are editable (See the Enhancements section for a discussion on dividing the client configuration into local and remote)

Settings

The settings page is only available when the client is connected. If it is not connected, then the client settings can’t be retrieved. If the client is connected, then a remote method is invoked that pulls the client configuration to the server and the values are displayed on the screen, as illustrated below:

36

Archives

The archives page allows the user to view the contents of their image archive. The archives are organized hierarchically by year, month, day, time and session. Images are shown as thumbnails. To limit the number of images displayed on the page at one time, only X (currently X=16) images are shown and the entire list of images for the chosen day are divided into TotalImages/X pages, with each page showing X images.

37

38

Technologies I-SPY encompasses a wide variety of cutting edge technologies

ecause I-SPY is a distributed architecture that uses a wide variety of tools, the technologies that are used for this project are vast. These include both client and server side libraries. As a middleware platform, the

very nature of J2EE is to integration. The developer faces a wide variety of design issues, and therefore technology choices. Below is a description of the Java technologies that are used in I-SPY. Advanced Java Technologies in Detail

Technology Description

Apache Tomcat 4.18 Tomcat is the servlet container that is used in the official Reference Implementation for the Java Servlet and JavaServer Pages technologies.

Other popular J2EE Application Servers include: • Sun One • IBM WebSphere • Bea WebLogic

URL: http://jakarta.apache.org/tomcat/

Apache Ant 1.5.1 Apache Ant is a Java-based build tool. In theory, it is kind of like Make, but without Make's wrinkles. This tool is platform independent and used to generate builds of both I-SPY and the I-SPY javadocs. URL: http://ant.apache.org/

Chapter

5

B

39

Java2 Enterprise Edition

Java2 Enterprise Edition (J2EE) technology and its component based model simplifies enterprise development and deployment. The J2EE platform manages the infrastructure and supports the Web services to enable development of secure, robust and interoperable business applications. The main goal of I-SPY is to learn J2EE, and is mandatory in J2EE systems. URL: http://java.sun.com/j2ee/

Java Advanced Imaging I/O Tools

The Java Advanced Imaging (JAI) I/O Tools provides reader, writer, and stream plug-ins for the Java Image I/O Framework and Image I/O-based read and write. Reader-writer plug-ins are supplied for the BMP, JPEG, JPEG 2000, PNG, PNM, Raw, TIFF, and WBMP image formats. The supplied streams and associated service providers use the New I/O APIs. URL: http://java.sun.com/products/java-media/jai/

Java Media Framework

The Java Media Framework (JMF) API enables audio, video and other time-based media to be added to Java applications and applets. This optional package, which can capture, playback, stream and transcode multiple media formats, extends the multimedia capabilities on the J2SE pl atform, and gives multimedia developers a powerful toolkit to develop scalable, cross-platform technology. In I-SPY this is the technology that interfaces with Video for Windows. In other words, this library makes available Media Resources, in particular a web-cam. URL: http://java.sun.com/products/java-media/jmf/

XML (Xerces Parser) Xerces is a high performance XML parser that comes standard with both Java2 1.4 and J2EE, There are Xerces libraries available for a wide variety of languages including C++ and Perl. URL: http://xml.apache.org

40

Java Server Pages Syntax Version 1.2

Java Server Pages (JSP) technology provides a simplified, fast way to create web pages that display dynamically-generated content. T he JSP specification, developed through an industry-wide initiative led by Sun Microsystems, defines the interaction between the server and the JSP page, and describes the format and syntax of the page. JSPs comprise one of the three foundational components of J2EE, the others being Servlets and EJBs. JSPs will comprise the largest component of the I-SPY J2EE development with hooks into Standard Java2. URL: http://java.sun.com/products/jsp

Servlets Java Servlet technology provides Web developers with a simple, consistent mechanism for extending the functionality of a Web server and for accessing existing business systems. URL: http://java.sun.com/products/servlet

EJB An E nterprise Java Bean is a server-side component that encapsulates the business l ogic of an application. EJB’s are most beneficial in large distributed systems. Although I-SPY can be developed without EJB’s, I-SPY may include an EJB for educational purposes. URL: http://java.sun.com/j2ee/tutorial

Java2 I-SPY uses many standard Java 2 libraries, such as RMI. Java2 is core to both J2EE and I-SPY. URL: http://java.sun.com

JBoss JBoss is an Open Source Application Server. JBoss is the container for EJBs (Enterprise Java Beans). URL: http://www.jboss.org

41

Future Enhancements

1. Improved Security There is several places in I-SPY that have been recognized as potential security holes. These are mainly the remote objects available on the client, as well as the EJBs residing on the server side. Both are a form of a RemoteObject, a Java middleware technology for convenient distributed programming. The distributed objects in I-SPY are of great value, but they also have special security issues must be thoroughly addressed before the system can be considered completely secure.

2. P2P Image Transfer

This enhancement will take the existing Image Transfer Protocol (ITP) and extend its functionality so that the server can be implemented on the client as well. By doing this, the images can be transferred directly to another client in a similar fashion as they are transferred to the server, thus producing a Peer-to-Peer communication model.

3. Real-time Streaming Video

Current only single frame images are captured from the capture device and sent across the network. This enhancement will send streaming video in place of the images. Due to the tremendous amount of bandwidth required to transmit video, a great deal of effort must be put towards minimizing the delay while maintaining adequate synchronization. A solution to this is the Real-time Transmission Protocol (RTP). RTP is a relatively new protocol developed specifically for dealing with synchronization issues of streaming video. A Java implementation of the RTP has been developed by Sun Microsystems and would integrate nicely into I-SPY.

4. Wireless Device Access

One of the main advantages of using J2EE infrastructure is that it provides convenient environment for developing wireless solutions. One of the original goals of the project was to eventually develop a wireless client that could access simple I-SPY services, such as viewing incoming images and remote management services (i.e.

Chapter

6

42

stopping/starting webcam). This would enable a user to view and control their webcam from anywhere and anytime.

5. Multiple Cameras

I-SPY is a system that can be used in many different contexts. One context in particular is a monitoring/surveillance system. In order to provide the necessary requirements for these scenarios, each user should be able to have various webcams associated with their account. I-SPY can handle numerous simultaneous clients, but they are currently all dealt with individually. For example, if a particular user wanted two webcams, they would have to configure each with a separate username/password and each webcam would have to be accessed from a separate web-client. The ideal situation would be that each user could have multiple webcams associated with their account, all of which could be viewed, maintained and controlled from a single web-client console.

6. Optional Public View of Incoming Images/Video

Currently only registered users are able to access their incoming image stream from a secured area within the I-SPY website. This enhancement would allow a user to specify that their incoming stream can be viewed publicly, either with or without a password. The "incoming image stream" was generalized to "incoming stream" because pending the addition of streaming video to the server, the video stream could also be made public as well.

7. RMI Socket Factory for Addressing Firewall Issues

The currently implementation of the remote client management services uses standard RMI. The problem here is that RMI by default does not have the ability to cross firewalls, however a solution is fairly simple and involves writing a custom RMI Socket Factory.

8. Improved Archiving

Archiving is an important feature to I-SPY because there is a tremendous amount of data coming in and being stored. The current archiving facility has minimal capabilities, and thus a new and improved archiving module would greatly enhance the overall functionality of the system. A new archiving module could include features such as compression and archiving of image, maintenance and cataloguing, etc.

9. Adaptive Synchronization of Console View

The console page on the client web access shows the incoming images. The page is refreshed at some given interval. Determining a correct refresh rate is difficult. One solution is to use the same interval that is used between image captures on the client side. However, due to network transmission times and other inherent delays, strictly using this time will still result in synchronization problems. Therefore one solution is to use an "Adaptive Synchronization" approach wherein each time the page is refreshed, the system can check to see if this is truly a

43

new image. If the image is new, then it can decrease the refresh rate, since it must be slower than the incoming image rate. Similarly, if the image is the same one, then the system can increase the refresh rate. By doing this, the system is continuously refining its refresh rate to try and adapt to the current network conditions and thus providing the best possible synchronization between refreshing the web page and the actual incoming images.

10. Improved Remote Settings Management

Currently the entire client configuration can be sent over the network to the web tier. Changes can be made there and the configuration is then sent back. Although consistency can be checked internally, it would be a valuable improvement to separate out parts of the client configuration that can’t be modified remotely.

11. Intelligent Network Sensing

I-SPY current has two network modes, Online and Offline. If the network is done, I-SPY can be put into Offline mode, meaning images are stored locally temporarily. However if the network comes up again the client will not attempt to re-connect. This enhancement would allow a user to specify that the client would notice the network was up, and automatically connect and resume in Online mode.

12. Intelligent Softwa re Motion-Sensing

This tool would allow a user to specify that I-SPY would not start capturing images/video until a movement was detected. This could be implemented in a primitive fashion using pixel-point processing to determine if a captured image was relatively different from the last, thus indicating a "movement".