VideoForgeVideoForge

An image processing systemAn image processing system

ContentsContents

Current state of the artCurrent state of the artIntroductionIntroductionTiersTiers

BaseBase CoreCore ModulesModules

User interfaceUser interface Interactive modeInteractive mode Scripting Scripting

universal interfaceuniversal interfaceprimary implementation COM scripting objectprimary implementation COM scripting object

Implementation templatesImplementation templates ControllersControllers AlgorithmsAlgorithms

Current state of the artCurrent state of the art

LibrariesLibraries Loki, BoostLoki, Boost Direct Show™, ACML, IPL(IPP), OpenCV, Microsoft Direct Show™, ACML, IPL(IPP), OpenCV, Microsoft

Vision SDK, ImageMagick, …Vision SDK, ImageMagick, …

ApplicationsApplications GraphEdit (part of the DirectShow)GraphEdit (part of the DirectShow) PhotoshopPhotoshop ImageJImageJ ImageForgeImageForge Adobe Premiere, …Adobe Premiere, …

IntroductionIntroduction

System for video and live-stream image System for video and live-stream image processingprocessing

Platform independentPlatform independent ANSI C++ANSI C++

Easy-to-do computer visionEasy-to-do computer vision

Three tier applicationThree tier application Base (Serialization, Communication, Logging)Base (Serialization, Communication, Logging) Core (Module management, Memory management)Core (Module management, Memory management) Modules (Devices, Algorithms,)Modules (Devices, Algorithms,)

BaseBase

SerializationSerialization Binary vs. textual storageBinary vs. textual storage Automatic vs. manual storageAutomatic vs. manual storage Type lists, storing of arrays, classes …Type lists, storing of arrays, classes …

Variable reflectionVariable reflection offers the possibility of calling the variable “by offers the possibility of calling the variable “by

name”name” m_plugin.FindVar<int>(“varName”);m_plugin.FindVar<int>(“varName”); m_plugin.GetVarNames(…);m_plugin.GetVarNames(…);

User interfaceUser interface

CommunicationCommunication Scripting possibilityScripting possibility User interface (Interactive mode)User interface (Interactive mode)

Types of communicationTypes of communication Actions – non-parametric methodsActions – non-parametric methods Properties – like variables, accessor methodsProperties – like variables, accessor methods Methods – methods with parametersMethods – methods with parameters Events – (e.g. onClose, …)Events – (e.g. onClose, …)

CoreCore

Module and plug-in managementModule and plug-in management EnumerationEnumeration RegistrationRegistration

Memory managementMemory management Memory (de-)allocationMemory (de-)allocation Correct memory handling for erroneous Correct memory handling for erroneous

modulesmodules

Actions (from plug-ins and core itself)Actions (from plug-ins and core itself) User interface handlingUser interface handling

Modules and plug-insModules and plug-ins

Module – fileModule – file Contains plug-in implementationsContains plug-in implementations compiled-in or represented by DLLcompiled-in or represented by DLL

Plug-insPlug-ins SourcesSources

Feeds data into chainFeeds data into chain AlgorithmsAlgorithms

Transforms data, somehowTransforms data, somehow Special plug-insSpecial plug-ins

usually compiled : persistence, etc.usually compiled : persistence, etc.

Image & FrameImage & Frame

similar to IPL IMAGE structuresimilar to IPL IMAGE structure

contains:contains: resolutionresolution ROI (rectangle of interest)ROI (rectangle of interest) MediaTypeMediaType

color mode (rgb, YUV, etc.) color mode (rgb, YUV, etc.)

bpp (=color depth)bpp (=color depth)

FrameFrame time information (start, stop time, fps)time information (start, stop time, fps) previous & next frame reference (=pointer)previous & next frame reference (=pointer)

Building the graphBuilding the graph

Pipeline buildingPipeline building Interactive mode vs. non-interactive modeInteractive mode vs. non-interactive mode

Device (Controller = Device enumerator)Device (Controller = Device enumerator)

Algorithms (Algorithm chain = Composite algs)Algorithms (Algorithm chain = Composite algs)

System windowsSystem windows

WindowsWindows show graphical informationshow graphical information part of a particular pluginpart of a particular plugin

Bitmap windowsBitmap windows shows contents of an image buffershows contents of an image buffer

Vector windowsVector windows usually used to show some information in the imageusually used to show some information in the image applied as a second layer onto the bitmap windowapplied as a second layer onto the bitmap window

Window eventsWindow events OnClose, …OnClose, …

Templates (1/3)Templates (1/3)

Application templateApplication template UI applicationUI application Console applicationConsole application

Module templateModule template default implementationdefault implementation rare modificationrare modification implements a list of plug-insimplements a list of plug-ins

Templates (2/3)Templates (2/3)

Controller templateController template Implement source enumerationImplement source enumeration Implement data feedImplement data feed Possibly implement UIPossibly implement UI

Algorithm templateAlgorithm template Implement data transformationImplement data transformation Possibly implement UIPossibly implement UI User must implement 1 methodUser must implement 1 method

ConclusionConclusion

We proposed system for implementation of We proposed system for implementation of image processing and computer vision tasksimage processing and computer vision tasks as platform-independent as possible (ANSI C++)as platform-independent as possible (ANSI C++) consists of (lightweight) core and pluginsconsists of (lightweight) core and plugins offers a user interactive, scriptable user interfaceoffers a user interactive, scriptable user interface offers various templates for fast implementation offers various templates for fast implementation

The system is a part of various grantsThe system is a part of various grants Grant UKGrant UK APVV GrantAPVV Grant

Future workFuture work

Finalize implementation of the core with the Finalize implementation of the core with the basic modules (12/05)basic modules (12/05) basic image processing operationsbasic image processing operations

Implementation of basic modules of 0(1/06)Implementation of basic modules of 0(1/06) working set optimizationworking set optimization

Implementation of special modules (06/06)Implementation of special modules (06/06) Feature trackingFeature tracking Cooperation with avatarsCooperation with avatars Statistics – classifier trainingStatistics – classifier training