beyond real-time video surveillance analytics with gpuspanoptes real-time and beyond video event...

[email protected] – www.intuvisiontech.com

Beyond real-time video surveillance

with GPU accelerated Panoptes

Dr. Sadiye Guler intuVision, Inc. August 14, 2013

GeoInt Accelerator • Platform to start developing with GPUs

• GPU access, GPU-accelerated apps and

libraries

• Register to learn more:

http://www.nvidia.com/geoint

Webinar Feedback Submit your feedback for a chance to win Tesla K20 GPU* https://www.surveymonkey.com/s/intuvision

* Offer is valid until September 1st 2013



https://www.surveymonkey.com/s/intuvision


intuVision, Inc. Proprietary

Outline

• Introduction

• Intelligent Video Surveillance Background

• intuVision Approach

• CPU vs CPU+GPU

– Utilizing GPU’s in Video Analysis

– Benefits in Performance and Accuracy

• OpenCV CUDA Advantage

• intuVision Panoptes System

• Concluding Remarks


Outline

• Introduction

• Intelligent Video Surveillance Background • intuVision Approach

• CPU vs CPU+GPU







Intelligent Video Surveillance: Promise

Intelligent analysis turns data into knowledge

Data collections

knowledge/ intelligence

information

Intelligent Video Surveillance … turns video cameras into

smart sensors…

Stacks of HD’s, reams of tapes

Walls of TV monitors

Decisions

Actions


If an image is worth a thousand words, a second of video may have 15-60 frames…1000 x framerate… worth a lot more

Once the frame contents are extracted into metadata it is more efficient to store/index/search

Intelligent Video Surveillance: Premise


Object motion and appearance features

Intelligent Video Surveillance

Object trajectories, Object classes

Higher level analyses

Activities, events, anomalies

Real time Alerts, Warnings / Post Analysis Investigation


Video Object Detection & Tracking Basics

• Background Model and Background Subtraction – Thresholded difference from a background model

– Different background models, Gray Scale, RGB color, Advanced color, Mean/Median, Mixture of Gaussians

– Provides approximate object silhouette

• Object detection and Tracking – Identifying foreground object pixels in each frame

– Correlating objects from frame to frame

– Classification of foreground objects (e.g. person, vehicle, etc.)

• Frame differencing – Thresholded difference of consecutive frames

– Provides a rough object region for moving objects

– Computationally inexpensive

– Stopped objects are no longer detected


Outline

• Introduction


• intuVision Approach • CPU vs CPU+GPU







intuVision Video Tracking Framework

Communications

Peripheral Tracker

Vision Tunnels

Stationary Object Layer

Objects

Tracks

Objects

BG Model

SO Model

Video

Input

Communications

Scene Description Layer

Peripheral Tracker

Object Layer

Stationary Object Layer

Video

Input

• Layered tracking approach inspired by human Multiple Object Tracking process

• Fast Peripheral Vision- Spatially based quick glance of the scene

– Fast but coarse detection and tracking of objects

• Object Tunnels- Object based focus for the tracked targets

– Detailed analysis and object classification

• Scene Description Layer

• Pixel, Edge and Noise Background Models

• Stationary Object Layer

• Detection of stopped or intermittent objects

• Task dedicated “as needed” processing

• Computational efficiency


Peripheral Tracking Layer

Stationary Object layer




Video Tracking Layers




Stationary Object LLayer




Object Layer

Object Object Object Object

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0 5 10 15 20 25 30 35




Input Frame

Background Model

Peripheral motion

Noise

Shadow pixels

Foreground object pixels



Video Event Detection

Input Frames

Intermediate Representation

Event Detection Relative motion and spatial states of detected objects

Classification of detected objects


Parallelizable tasks in intuVision framework

• Detecting short term Peripheral object motion

• Maintaining a long term background model for every pixel

– Update background mean and standard deviation

– Support for GrayScale, RGB and La*b* color models for background

• Determining foreground pixel probability validated by Peripheral motion compare Background with each new frame

• Detecting stationary and intermittent objects

• Maintaining False Motion Models to account for noise due to dynamic scene elements

• Applying morphological filters to foreground pixels


Outline

• Introduction



• CPU vs CPU+GPU – Utilizing GPU’s in Video Analysis






CPU vs GPU

• CPU architecture – few general purpose processors capable of multithreading

– optimized for instruction level parallelism

• GPU architecture – many small specialized processors executing Single Instruction on

Multiple Data (SIMD)

– optimized for data

level parallelism

core

[0,1]

core

[0,2]

core

[0,6]

core

[1,1]

core

[1,2]

core

[1,6]

core[3,

1]

core

[3,2]

core

[3,6]

core

[0,0]

core

[1,0]

core

[3,0]

core

[0,3]

core

[1,3]

core

[3,3]

core[0,

4]

core

[1,4]

core

[3,4]

core

[0,5]

core

[1,5]

core

[3,5]

core

[2,1]

core

[2,2]

core

[2,6]

core

[2,0]

core

[2,3]

core

[2,4]

core

[2,5]

core

[0,7]

core

[1,7]

core

[3,7]

core

[2,7]

G

P - 0

U

G

P - 1

U

G

P - 2

U

G

P - 3

U

Reduced computing time

Reduced hardware costs

Reduced power consumption


All pixel based operations are independent of their neighbors and

can be performed in parallel on the GPU

– Reduced computing time - More frames processed per second

– Reduced hardware costs - Low-end computers can be boosted with GPU’s

Video Analysis with GPU Acceleration

GPU

[0,1]

GPU

[0,2]

GPU

[0,6]

GPU

[1,1]

GPU

[1,2]

GPU

[1,6]

GPU

[3,1]

GPU

[3,2]

GPU

[3,6]

GPU

[0,0]

GPU

[1,0]

GPU

[3,0]

GPU

[0,3]

GPU

[1,3]

GPU

[0,4]

GPU

[1,4]

GPU

[3,4]

GPU

[0,5]

GPU

[1,5]

GPU

[3,5]

GPU

[2,1]

GPU

[2,2]

GPU

[2,6]

GPU

[2,0]

GPU

[2,3]

GPU

[2,4]

GPU

[2,5]

GPU

[0,7]

GPU

[1,7]

GPU

[3,7]

GPU

[2,7]


CPU only CPU + GPU

CPU

GPU

[0,1]

GPU

[0,2]

GPU

[0,7]

GPU

[1,1]

GPU

[1,2]

GPU

[1,7]

GPU

[3,1]

GPU

[3,2]

GPU

[3,7]

GPU

[0,0]

GPU

[1,0]

GPU

[3,0]

CPU

Time

High processing fps, left object detected in time


• In the GPU architecture the data parallel portions of an application are executed as kernels in parallel

• The CUDA processing model uses a grid of 1D, 2D or 3D blocks with each block running multiple threads

• In this model it is important to minimize the control flow divergence as each thread executes the same kernel and any divergence results in the slowing down of all the threads

• A group of 32 threads is defined as warp in the CUDA model and it corresponds to the smallest executable unit of parallelism on the GPU device

GPU Instruction Execution


GPU Memory to Remember

• The CUDA memory model contains multiple memory spaces: – Thread, local, block, shared, global, constant and texture memory spaces

– Global, constant and texture memory spaces are persistent across kernels

– Global, local and texture memory have the greatest access latency

intuVision CUDA implementation employs:

• Reduced memory access and improved memory access patterns

• Memory coalescing – Enforces consecutive threads of a warp to concurrently request consecutive

logical addresses from global memory to minimize the global memory access

• Use of CUDA textures – cached unlike the global memory

Side by Side

Vehicle Counting

but far apart… intuVision, Inc. Proprietary

GPU Performance Benefit Example

Video Resolution CPU (msec) GPU (msec)

GrayScale RGB La*b* GrayScale RGB La*b*

640 x 480 16.50 21.05 58.05 10.66 10.21 12.82

1280 x 720 39.06 56.80 184.23 16.10 16.07 23.63

Use of the GPU enables faster than real time processing with the standard RGB model and the more computationally intensive advanced color La*b* model

Off-the –shelf hw platform: CPU – Intel Core i7-3770 (4 Cores)

GPU – NVIDIA GTX690 (3092 Cores)

Better than real-time


GPU Performance Benefits in Numbers

CPU+ GPU processing speed up factor over CPU

0

1

2

3

4

5

6

7

8

9

640 x 480 1280 x 720

Grayscale

RGB

La*b*

Video Resolution

GP

U-C

PU

sp

ee

du

p fa

cto

r

Intel Core (4) i7-3770 NVIDIA GTX690


Full scale Video Analysis Video Object Detection

L*a*b* Color Background Model RGB Background Model

Grayscale Background Model

Missed pixels

Shadows

GPU Accuracy Benefits


GPU Accuracy Benefits (before)

Large area monitoring with wide angle view HD camera

Traditional CPU based processing cannot maintain a real time frame rate

– Results in background model deterioration and missed detections


GPU Accuracy Benefits (after)

Large area monitoring with wide angle view HD camera

GPU based processing of the video correctly learns the background

– Results in reduced missed detections, better motion history improved tracking



Outline

• Introduction



• GPU vs CPU


– GPU Benefits in Performance

• OpenCV CUDA Advantage • intuVision Panoptes System



intuVision use of OpenCV CUDA

• More than 250 optimized functions

• Supports filtering, feature extraction, object detection, optical flow, etc.

– Computation intensive tasks like object detection can be performed in real time

• Haar Cascade Classifier – 6x speed up

• Allows for quick prototyping of algorithms and determining the GPU speedup


Clutter and dynamic texture detection

Without clutter and dynamic texture removal

With clutter and dynamic texture removal

Input Image with boat&wake detection

Foreground Image


Example use of OpenCV: Clutter and dynamic texture removal filter

• Identify techniques for clutter and dynamic texture detection

• Validate a prototype using OpenCV CUDA implementations find out GPU speedup potential

• Prototype optimization

– Develop optimized version of algorithm components

– Replace off the shelf algorithms with custom implementation

• GPU implementation of clutter and dynamic texture detection results in more than 2.5x speedup


Clutter and dynamic texture removal

Without clutter and dynamic texture removal

With clutter and dynamic texture removal


Outline

• Introduction



• GPU vs CPU


– GPU Benefits in Performance


• intuVision Panoptes System • Concluding Remarks


Panoptes Real-Time and Beyond Video Event Monitoring

Panoptes “the all seeing”

Argus Panoptes –the God with 1000 eyes

Event detection and logging , alarm schedules, alarm metadata and triggers

Faster than real-time processing of archived video (up to 120fps)

Video from live or archive streams

BW, Color and IR

CIF to HD resolutions

Panoptes Event:

Wrong Direction

9:38 am Jan 15, 2010


Panoptes Intuitive User Interface

Color coded List of Detected Alarms

Current Alarm Detection marked with a flashing red frame

IR Camera Monitoring

Selected Alarm Event Details


Panoptes

Object Classification User trainable object

classification

Automatically collected

training samples

Robust models for

people and vehicles

Learned models apply to

similar views Training Samples

Person Vehicle Group


Panoptes Events

Abandoned Object Objects left unattended for a period of time

Activity Objects detected in a Region of Interest

Crowd Density People gathering for a set period of time

Enter/Exit An object enters/exits the view from a specified area

Idle Object A moving object coming to stop for a duration

Line Crossing An object moving passed over a specified line


Panoptes Events

Object Taken Objects moved from marked regions

Object Counting Counting objects in a Region of Interest

Perimeter Intrusion A moving object entering into a marked zone

Speeding Object An object moving too quickly between set lines

Wrong Way Movement in the specified wrong direction

Smoke & Fire Detection Detecting sections of a scene with smoke or fire


Panoptes Counting Events

Count any Panoptes Event

Use with any object type

Person

Vehicle

Animal

Boat

Etc.

Easy-to-use count options

Overlay on camera view

List in the object counting

window


Panoptes Compound Events

Compound Events:

Specified events happening in combination over camera(s)

Wrong Direction in Camera-1 Stopped vehicle in Camera-2

AND


Camera-2

Idle Vehicle

Intrusion

Compound Event

Panoptes

Floor Plan Camera Monitoring

Monitor cameras

from an intuitive

floor-plan layout

Observe camera

interactions at a

glance

Receive notifications

and respond quickly

as events unfold


Panoptes Compound Event

Get-away car & indoor office activity after hours.


Outline

• Introduction



• CPU vs CPU+GPU







Summary

• Video Content Analysis continues to be a fast growing technology area

with lots of potential and applications

• Accuracy reliability, scalability, performance and robustness matter!

• GPU’s offer low cost options for increased performance and scalability

– With increased performance more complex algorithms can be run in real-

time (and beyond) resulting in increased accuracy

• intuVision Video Analytics are GPU accelerated

– Performance gains up to 12x are obtained with addition of Nvidia GPU

cards to off-the-shelf low-end hardware platforms

– Optimization to newer GPU platforms is ongoing


Video Content Analysis: Practice

data knowledge/

intelligence information incomplete

uncertainty

noise

Decisions

Actions

• Several interesting problems remain to be solved in extracting video content:

• Multi camera environments, understanding events at different granularity…

• Also several not-so-interesting problems remain to be solved to have robust

systems:

• Illumination variations, shadows, occlusions, camera jitter, varying frame rates…

Human

in the

loop


Thank you!

Dr. Sadiye Guler intuVision, Inc.

www.intuvisiontech.com

Surveillance bee

Upcoming GTC Express Webinars

Register at www.gputechconf.com/gtcexpress

August 15 - CUDA 5.5 Production Release: Features Overview

September 5 - Data Discovery through High-Data-Density Visual

Analysis using NVIDIA GRID GPUs

September 10 - Virtualizing Tough 3D Workloads with VMware

Horizon View and NVIDIA Technologies

September 12 - Guided Performance Analysis with NVIDIA

Visual Profiler

September 17 - ArrayFire: A Productive GPU Software Library

for Defense and Intelligence Applications

GTC 2014 Call for Submissions

Looking for submissions in the fields of

Science and research

Professional graphics

Mobile computing

Automotive applications

Game development

Cloud computing

Submit by September 27 at www.gputechconf.com

GeoInt Accelerator • Platform to start developing with GPUs

• GPU access, GPU-accelerated apps and

libraries

• Register to learn more:


Webinar Feedback Submit your feedback for a chance to win Tesla K20 GPU* https://www.surveymonkey.com/s/intuvision

* Offer is valid until September 1st 2013





beyond real-time video surveillance analytics with gpuspanoptes real-time and beyond video event...

Documents