UofC Large Display Framework

Part II: Toolkit & Usage

Fabrício AnastácioSeptember, 2007

Outline

• Part I:– Basic Concepts– Objectives– New Structure– I-Buffer Layer– Framework Layer– Framework Classes

• Part II:– Toolkit Layer– Interface with the

Framework– Extending the

Framework Classes– Types of Strategies– Some Toolkit Strategies– Creating an Application– Conclusion

Toolkit Layer

• Separated DLL, depends on the Framework and IBuffer DLLs

• Provides a set of classes that can be used by an application

• Uses Qt, SBSDK, and OpenGL by default

• Allows an application to focus on content creationI-Buffer

Large Display Framework

Component Toolkit

Specific Application

Interfacing with the Framework

IBuffer

TouchIndicator

LargeDisplayManager VisComponentStrategy

VisComponentStateCompositeNode VisComponent

IBufferProxy

FrameTimer

VectorTemplate

Point3f

Extending the LargeDisplayManager class

• Functions to implement (pure virtual in the parent):– void initComponents()– void initTouchIndicators()– unsigned long pickComponent(unsigned int x, unsigned

int y)– unsigned long pickContainer(unsigned int x, unsigned

int y, unsigned long excludeComponentId)

• Optionally:– void resize(unsigned int width, unsigned int height)

• A default manager (DefaultManager class) is provided, which implements all these methods but initComponents()

LargeDisplayManager initComponents()

• It is where the root and the other components initially available/visible to the user are created

• Creating a component:– Instantiate a VisComponent object in a pointer:

• VisComponent* component = new VisComponent();

– Set the component properties, like position, width, height, rotation angle, etc.

– Create strategies for the component:• RNTStrategy* rnt = new RNTStrategy();

– Push the strategies into the component:• component->pushStrategy(rnt);

– If the component is a leaf, set its leaf flag:• component->setLeaf(true);

– Add the component to the manager:• addComponent(component);

LargeDisplayManager initTouchIndicators()

• Creates a TouchIndicator object for each user:– RoundTouchIndicator* indicatorUser1 = new RoundTouchIndicator();

• Adds it to the list of touch indicators in the manager:– touchIndicators.push_back(indicatorUser1);

LargeDisplayManager pickComponent()

• Provides the rendering API-dependent routine for picking a component

• It takes the x and y values of the screen coordinates (with origin in the lower left corner) and returns the ID of the picked component or 0, if none was picked

• The default implementation:– Renders all the components in picking mode (i.e., in a specific

color obtained from the component’s ID) in the back buffer– Reads the color of the pixel at the position given as parameter– Converts the color back to a component ID value and returns it

• It is should be called by the framework and not by the application

LargeDisplayManager pickContainer()

• Provides the rendering API-dependent routine for picking a container

• It takes the x and y values of the screen coordinates (with origin in the lower left corner) and the ID of a container to be excluded from rendering (if any); and returns the ID of the picked container or 0, if none was picked

• It is implemented in the same way as the pickComponent() method with the difference that, instead of rendering all components, it renders only the containers (with the specified exception) in picking mode

• It is should be called by the framework and not by the application

LargeDisplayManagerresize()

• Defines what should be done when the application is resized

• Should only be over-ridden if additional behavior is necessary

• The LargeDisplayManager implementation already takes care of updating the values for the screen dimensions and forwarding the call to the root component

• In the DefaultManager, the root component is also centralized in the resized screen

Extending the TouchIndicator class

• Functions to implement (pure virtual in the parent):– void drawIndicator()– void drawExtent()

• These functions implement the rendering API-specific routines that draw the indicator and its extent (e.g., its bounding rectangle)

• The toolkit provides a circular touch indicator with color gradient in the RoundTouchIndicator class

Extending the FrameTimer class

• Functions to implement (pure virtual in the parent):– unsigned long getCurrentTime()– double calculateFrameRate()

• The getCurrentTime() function is only an interface for getting the value of the current system time

• The calculateFrameRate() function returns the number of frames per second calculated from the average frame interval duration (obtained from the superclass) and the time unit used by the specific OS

• The toolkit provides a MS Windows class (WindowsFrameTimer) that uses the GetTickCount() method from the Windows API and calculates the frame rate by converting clock ticks to seconds

Extending the VisComponentStrategy class

• Functions to implement (pure virtual in the parent):

– void initProperties()– void draw(const std::vector<unsigned long>& selectedIds)– void drawForPicking()– void pressIn(unsigned int x, unsigned int y) – void pressOut(unsigned int x, unsigned int y)– void drag(unsigned int x, unsigned int y) – void release(unsigned int x, unsigned int y)– void pointIn(unsigned int x, unsigned int y)– void pointOut(unsigned int x, unsigned int y)– void resize(unsigned int width, unsigned int height) – bool drop(bool parentChanged)– void readPassiveBuffers(const std::vector<unsigned int>& types)– void readAllPassiveBuffers()– void readAllPickingPassiveBuffers()

VisComponentStrategy initProperties()

• Creates and initializes active, passive, and/or private buffers for a component

• Creating an active buffer:– Instantiate a IBufferProxy object:

• IBufferProxy* bufferProxy = new BufferProxy();– Set the buffer type:

• bufferProxy->setType(BufferConstants::SCALE_BUFFER);– Create the buffer:

• IBuffer<float>* buffer = new IBuffer(BufferConstants::SCALE_BUFFER, nChannels, component->getWidth(), component->getHeight(), component->getWidth(), component->getHeight());

– Fill the buffer (a private function can be created for that):• fillScaleBuffer(buffer);

– Set the buffer to the proxy object:• bufferProxy->setBuffer(buffer);

– Add the buffer to the component:• component->addActiveBuffer(bufferProxy);

VisComponentStrategy initProperties()

• Creating a passive buffer:– Instantiate a IBufferProxy object:

• IBufferProxy* bufferProxy = new BufferProxy();– Set the buffer type:

• bufferProxy->setType(BufferConstants::SCALE_BUFFER);– Set the buffer of the proxy object as null:

• bufferProxy->setBuffer(NULL);– Add the buffer to the component:

• component->addPassiveBuffer(bufferProxy);

• For private buffers, the IBufferProxy object is not necessary and the IBuffer<> instance is a private (or protected) instance variable. It is not necessary to add it to the component either

VisComponentStrategydraw() & drawForPicking()

• These functions implement the rendering API-specific routines to draw a component (or its decoration) for displaying and picking

• In the toolkit, OpenGL is used as the rendering API

• The draw() method has the list of selected IDs as parameter so that it can apply different rendering rules to selected components (highlighting)

• In the drawForPicking() function, the component (or decoration) should be rendered in the color obtained from its ID value:

unsigned char pickingColor[3] = { 0, 0, 0 };component->getIdColor(pickingColor);

• To improve performance, its strongly suggested that the rendering routines should be implemented as OpenGL display lists, which are initialized by the strategy constructor and called by the drawing methods

VisComponentStrategyDevice Functions

• pressIn(): called when the device is pressed inside the bounds of the associated component

• pressOut(): called when the device is pressed outside the bounds of the associated component

• drag(): called when the component is dragged by the device

• release(): called when the device is release from the associated component

• pointIn(): called when the device cursor is over the associated component

• pointOut(): called when the device cursor is not over the associated component

VisComponentStrategy resize()

• The resize() function is more commonly used to propagate the resizing of an associated component to the active or private buffers managed by the strategy

• For example:

if (component) { IBufferProxy* bufferProxy = component->getActiveBufferByType( BufferConstants::SCALE_BUFFER); if (bufferProxy && bufferProxy->getBuffer()) { IBuffer<float>* buffer = (IBuffer<float>*) bufferProxy->getBuffer(); buffer->resize(width, height, width, height); fillSizeBuffer(buffer); }}

VisComponentStrategy drop()• The drop() function defines what should be done when the associated

component is “dropped” inside a container

• It usually defines some “decorative” behavior such as triggering animations

• The parameter parentChanged indicates if the component’s parent (container) changed with the “dropping”

• It should return true if the associated component is under a container for which an animation is triggered

• An animation for adding a component in a container is triggered by calling the animateAdding() method in the VisComponent class, defining the type of the buffers that should be read by the component (for state changes) and what states should be left unchanged by the animation:component->animateAdding(bufferTypes,

VisComponent::POSITION | VisComponent::ROTATION_ANGLE);

VisComponentStrategyBuffer Reading Functions

• readPassiveBuffers(): goes over the list of buffer types passed as parameter and checks if it has any of the types of the passive buffers watched by this strategy. If it does, these buffers are read for the current component position

• readAllPassiveBuffers(): reads all the passive buffers watched by this strategy for the current component position

• readAllPickingPassiveBuffers(): reads all the passive buffers watched by this strategy for the current component position that are relevant for picking rendering

Buffer Reading Example

• Case I:

IBufferProxy* bufferProxy = component-> getPassiveBufferByType(BufferConstants::SCALE_BUFFER);if (bufferProxy && bufferProxy->getBuffer()) { IBuffer<float>* buffer = (IBuffer<float>*) bufferProxy->getBuffer(); double bufferX = 0, bufferY = 0; bufferProxy->getOwner()->convertGlobalToBufferCoords( component->getPosition().x, component->getPosition().y, bufferX, bufferY); if (buffer->isInsideActualBuffer(bufferX, bufferY)) { float s = buffer->getValue(bufferX, bufferY); component->setScaleFactor(s); } }

Buffer Reading Example

• Case II:

if (component) { try { float s = component->getPassiveBufferValue<float>( BufferConstants::SCALE_BUFFER, component->getPosition().x, component->getPosition().y); component->setScaleFactor(s); } catch (std::exception e) {}}

• Case II is nicer than Case I, but, in initial evaluations, made the test application approximately 10 times slower!

Types of Strategies• Depending on their purpose, strategy classes can be

classified as:

– Behaviors: provide an interaction behavior to a component• Ex.: RNT, translation, and tossing.

– Containers: have properties that affect the components that are inside them

• Ex.: Friction surfaces and interface currents.

– Observers: allow a component to respond to a certain type of buffer• Ex.: Scale observer, rotation observer, and current observer.

– Drawing: define how to draw a component or how it is decorated• Ex.: Images, borders, and resizing handles

– GUI: transform components into widgets• Ex.: creators and destroyer buttons

Some Toolkit Strategies• ImageStrategy (Drawing):

– Draws a component as a quad with texture

– Implements only the draw() and drawForPicking() methods

– The OpenGL texture name should be passed to the constructor

• BorderStrategy (Drawing):

– Draws a border around the component’s bounds

– If the component is selected, draws the border in a different color

– Implements only the draw() method

Some Toolkit Strategies• ScaleObserverStrategy (Observer):

– Changes the scale factor of the associated component with a value read from scale buffer

– Implements the initProperties(), drop(), readPassiveBuffers(), readAllPassiveBuffers(), and readAllPickingPassiveBuffers() methods

– A passive scale buffer is added by this strategy to the component in the initProperties() method

– Animation of the scaling process is triggered by the drop() method

Some Toolkit Strategies

• RNTStrategy (Behavior):

– Provides RNT (Rotation ‘N Translation) behavior to the component

– Implements the draw(), pressIn(), drag(), and release() methods

– The draw() function only renders the translation circle inside the component (which can be disabled)

– The pressIn() function checks a button buffer (if present) before activating the behavior

Some Toolkit Strategies• TossableStrategy (Behavior):

– Allows a component to be tossed around

– Implements the initProperties(), draw(), pressIn(), drag(), release(), readPassiveBuffers(), readAllPassiveBuffers(), and readAllPickingPassiveBuffers() methods

– Looks for a friction buffer. If one is not found, user-defined default friction values are used

– The draw() method is used to update the component position for every frame while the tossing lasts (no actual rendering)

– After the tossing is over, the component is added to the manager’s update list to check if it ended up inside of a container and do the proper adjustments

Some Toolkit Strategies• MagnifierLensStrategy (Behavior):

– When the associated component is moved over a container that has a scale (active) buffer, it adds a value to the elements of this buffer at its position

– Implements the initProperties(), pressIn(), drag(), release(), and resize() methods

– Has a scale buffer as a passive buffer (although it writes to it) and has a private buffer filled with the values used to write to the scale buffer

– To avoid leaving the scale buffer with the magnified values when the lens component is no longer at that position, when the component is moved the magnified values at the previous position are erased

– The resize() method changes the dimensions of the private buffer

– This strategy can be extended to have different shapes (e.g.: QuadMagnifierLensStrategy and RoundMagnifierLensStrategy) and different types of magnification (e.g.: GaussianLensStrategy)

Some Toolkit Strategies• CreatorStrategy (GUI):

– Extends the ImageStrategy class and needs a texture name as parameter of the constructor

– Provides a base class for creating components that, when pressed, create another component

– Implements only the pressIn() method

– Defines a createComponent() pure virtual method that should be implemented by subclasses to provide the code for creation of a specific component

• DestroyerStrategy (GUI):

– Extends the ImageStrategy class and needs a texture name as parameter of the constructor

– Implements only the draw() method, adding code to remove and destroy all the current children of the associated component

Some Toolkit Strategies• CurrentBeltContainerStrategy (Container):

– Provides an implementation of the interface currents

– It inherits from the BeltContainerStrategy, which in turn inherits from the BoundedContainerStrategy

– The BoundedContainerStrategy provides a container with boundary defined by a spline curve; it has its own gradient border; and has a button buffer for controlling interaction

– The BeltContainerStrategy adds an inner boundary to the container

– The CurrentBeltContainerStrategy adds active buffers and their filling methods for absolute size, orientation, and movement direction to the container

– This hierarchy implements the initProperties(), draw(), drawForPicking(), pressIn(), drag(), release(), and resize() methods

Some Toolkit Strategies

• CurrentObserverStrategy (Observer):

– Provides a component with the ability to react to a current belt container

– Observes three different types of buffer (absolute size, orientation, and movement direction)

– Implements the initProperties(), drop(), readPassiveBuffers(), readAllPassiveBuffers(), and readAllPickingPassiveBuffers() methods

Creating an Application• Auxiliary libraries should be used (e.g.: Qt and SBSDK)

• A main class should be created, opening a QApplication with a QGLWidget

• The QGLWidget subclass can be defined by the user or inherited from the QGLSBToolkitWidget class

• The QGLSBToolkitWidget class provides an implementation of a QGLWidget and a CSBSDK2EventHandler

• It provides OpenGL initialization, resizing, and drawing; Qt mouse/keyboard and SmartBoard event handlers; and multi-monitor full-screen setup for Windows

• A subclass of LargeDisplayManager should be passed to the QGLSBToolkit; it can be an extension of the DefaultManager class over-riding only the initComponents() method

Example of initComponents()void MyManager::initComponents() {

float borderColor[4] = { 0.3, 0.5, 1.0, 1.0 };float selectedColor[4] = { 0.8, 0.6, 0.1, 1.0 };

rootComponent->setPosition(Point3f(400, 300, 0));rootComponent->setWidth(800);rootComponent->setHeight(600);FrictionSurfaceStrategy* friction = new FrictionSurfaceStrategy();rootComponent->pushStrategy(friction);rootComponent->getStrategy()->initPropertiesComponent();

VisComponent* componentCurrent = new VisComponent();componentCurrent->setPosition(350, 200);componentCurrent->setWidth(600);componentCurrent->setHeight(500);CurrentBeltContainerStrategy* currentBelt = new CurrentBeltContainerStrategy(150);componentCurrent->pushStrategy(currentBelt);currentBelt->initialize();addComponent(componentCurrent);

// A TextureManager (also from the toolkit) is created and// intialized with some textures here

Example of initComponents()

for (int i = 0; i < 50; i++) { VisComponent* c = new VisComponent(); c->setPosition(currentBelt->getRandomInsidePoint()); float dimension = obtainRandomFloat(150, 30); c->setWidth(dimension); c->setHeight(dimension); c->setRotationAngle(obtainRandomFloat(90) * TO_RADIANS_FACTOR); ImageStrategy* imageStrategy = new ImageStrategy(texManager.getRandomTextureName()); c->pushStrategy(imageStrategy); BorderStrategy* border = new BorderStrategy(); border->setColor(borderColor); border->setSelectedColor(selectedColor); c->pushStrategy(border); TossableRNTStrategy* rnt = new TossableRNTStrategy(); rnt->setColor(selectedColor); c->pushStrategy(rnt); CurrentAwareStrategy* currentAware = new CurrentAwareStrategy(); c->pushStrategy(currentAware); c->setLeaf(true); addComponent(c);}

}

Status Quo• Currently, the toolkit layer is provided in a DLL

(~40 classes) + some adjustments remaining

• Following steps:– Performance evaluation & Optimization– Further code clean-up– Documentation review– Beta testing & debugging– Simple example applications & tutorials– Toolkit layer expansion (if necessary)– Further refactoring

Special Interest

• Other rendering APIs:– Direct3D

• Other platforms:– MacOS– Linux

• Native usage of the set of DLLs by Java & C#

• Possible addition of a script layer as an alternative for simple content creation using the toolkit classes

Remembering: Objectives• Keep the same functionality

• Make the framework independent of the applications that use it

• Make the buffer data structure independent of the framework

• Separate specialized components from the framework and individual applications

• Make large display applications easier to create

• Keep a good performance

• Add some new features (maybe)

Previous Meeting• Dynamic definition of buffer types and button buffer

values– BufferConstants class was converted into BufferHelper (a

singleton class), providing a method to obtain/create replacements for the constants:

unsigned int getBufferTypeIdentifier(std::string description)unsigned int getButtonBufferValue(std::string description)

• Conversion of the TouchIndicator into a special type of component (with a “TouchIndicatorStrategy”)– Waiting definition of possible framework changes

• Definition of a generic event handling mechanism (with a more strict implementation of the Strategy Pattern)– Alternative I: partially use the current approach (for drawing &

buffers) and handle events in the application accessing the selected components

– Alternative II: bypass the current approach (Strategy) and extend the VisComponent class

End of Part II