autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_colorado...2016/08/23...
TRANSCRIPT
1
Autonomous robotic cameras
for collaborative target localization
(… and a privacy filter)
Andrea Cavallaro
@smartcamerassmartcameras
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
outline
2
From traditional camera views …
pedestrian tracking on MOT16 challenge dataset
X
Y
… to a mini-drone view
motion patterns
Trajectory clustering for motion pattern extraction in aerial videos
T. Nawaz, A. Cavallaro, B. Rinner
Proc. of IEEE Int. Conference on Image Processing (ICIP), Paris, October 27-30, 2014
3
Extraction of motion patterns
Multi-feature object trajectory clustering for video analysis [source code]
N. Anjum, A. Cavallaro
IEEE Transactions on Circuits and Systems for Video Technology, Vol. 18, Issue 11, November 2008
Trajectory clustering for motion pattern extraction in aerial videos
T. Nawaz, A. Cavallaro, B. Rinner
Proc. of IEEE Int. Conference on Image Processing (ICIP), Paris, October 27-30, 2014
Compensated trajectories
input trajectoriescompensated trajectories overlaid
on the scene mosaic
4
Motion pattern extraction
Mean-shift clustering with incremental bandwidth selection
clusters motion patterns
Multi-feature object trajectory clustering for video analysis [source code]
N. Anjum, A. Cavallaro
IEEE Transactions on Circuits and Systems for Video Technology, Vol. 18, Issue 11, November 2008
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
5
From single to multiple hovering cameras
The problem
Continuous estimation of the state of an object (target)
given a set of measurements (observations)
obtained from spatially distributed sensing nodes (cameras)
Measurements
State estimation
)zzz( Z N
k
2
k
1
kk
), x, Zf(Zx :k-:kkk 1011
1
kz
2
kz
3
kz
4
kz
6
Top view
Camera allocation for target tracking
• Constraints
– limited resources
– wireless communication
• packet errors
• packet losses
• Distributed tracking via camera grouping (coalitions)
– beneficial to distributed fusion
• quicker convergence
• greater tracking accuracy
– communication efficient
• comparable cost to decentralized fusion
• cheaper than distributed fusion without coalitions
7
Proposed coalition framework
Consensus protocols for distributed tracking in wireless camera networks
S. Katragadda, J. SanMiguel, A. Cavallaro
In Proc. of Information Fusion (FUSION), Salamanca, July 7-10, 2014
Coalition
candidate
announcement
Communication-
aware
coalition
formation
Distributed
fusion
(EIWCF)
[Katragadda2014]
Bridge
identification
Three-stage coalition formation
Stage 1 Stage 2 Stage 3
EIWCF: Extended Information
Weighted Consensus Filter
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015
Coalition candidate announcement
• After target detection
OR
Camera to join multiple coalitions
Announcement to neighbours
c1
c6 c7
c2
c5
c4
c3
measurement
prior estimate
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015
8
Communication-aware coalition formation
• For each candidate coalition
– information contribution to neighbours
• joining coalition
– contribute information for tracking
– traffic packet reception ratio
• not joining coalition
– not contributing information for tracking
– traffic packet reception ratio
– join a coalition if , where
U : utility
i : camera index
j : target index
- : excluding
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015
Bridge identification
• Distributed fusion requires a connected network
Bridge camera
Broadcast notification if ‘lonely’
Camera not in the coalition
c1
c6 c7
c2
c5
c4
c3
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015
9
Coalitions for target tracking
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
10
Tracking on demand: overview
• Hovering (static) cameras
– detect targets and form groups to fuse information
– send help requests to mobile cameras
• Mobile cameras
– select the target to track
– move to track the target jointly with hovering (static) cameras
static camera
mobile camera
Objective
• Collaboration with ‘on-call’ mobile cameras for
– prioritized target observation
– tracking accuracy
– energy efficiency
tracking
accuracy
prioritized
target
observation
energy
efficiency
Optimal controller for active tracking
Local robot-target assignment
Prioritized target tracking with active collaborative cameras
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
11
Challenges
• Limited on-board energy
– maintain tracking performance with smaller energy cost
• Limited communication range
– local decision making
Local target selection
request to
mobile cameras
after receiving request each robot selects a target to track
Prioritized target tracking with active collaborative cameras
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
12
Motion planning
• Model Predictive Control (MPC) [Findeisen2002]
– 𝑇ℎ: time horizon
– compute next robot state using the predicted state at 𝑡 + 𝑇ℎ
– minimize a weighted sum of cost functions: 𝐽1 and 𝐽2
𝐽1 aims to center target in FoV 𝐽2 aims to reduce energy [Liu2014]
cc
3 targets & 2 mobile cameras
Switching to track target
with higher priority 𝑜𝑗
𝑤𝑗: target 𝑜𝑗 with priority 𝑤𝑗
𝑐𝑖 receives a request to track target 𝑜𝑗
𝑜𝑗 𝑐𝑖𝑚𝑖𝑅
13
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
Team of flying cameras
1
4
3
2 d1,2
d1,4
y
x
z
xt(k)
Rt(k)
e1
e2
e3
b1,1
b2,1
b3,1
x1(k)
R1(k)
Objective: to achieve formation on target for monitoring and filming
14
Views of the target
Examples of flying camera views
target
A simple force-based model
goal force
interaction
forces
GPS sensing
positions of
the neighbors
position of the target
quadrotor
dynamics
Self-positioning of a team of flying smart cameras
F. Poiesi, A. Cavallaro
in Proc. Int. Conf. on Intelligent Sensors, Sensor Networks and Inf. Proc. (ISSNIP), Singapore, April 2015
15
Self-positioning of a team of flying smart cameras
F. Poiesi, A. Cavallaro
in Proc. Int. Conf. on Intelligent Sensors, Sensor Networks and Inf. Proc. (ISSNIP), Singapore, April 2015
How to follow a target without GPS?
velocities predictions
vision-based
inference
(position mapping)
video
analyticsdistributed
consensus
formation
maintenance
quadrotor
dynamics
• Noisy target detections could generate incorrect control
• Target could exit the fields of view
Distributed vision-based flying cameras to film a moving target
F. Poiesi, A. Cavallaro
in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept-Oct. 2015
16
• Video 2
Distributed vision-based flying cameras to film a moving target
F. Poiesi, A. Cavallaro
in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept-Oct. 2015
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
17
The (physical) risks for mini-drones …
Detection of fast incoming objects with a moving camera
F. Poiesi, A. Cavallaro
Proc. of British Machine Vision Conference (BMVC), York, September 19-22, 2016
Safe-point estimation for collision avoidance
Detection of fast incoming objects with a moving camera
F. Poiesi, A. Cavallaro
Proc. of British Machine Vision Conference (BMVC), York, September 19-22, 2016
18
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
Location-based privacy: geo-fencing
https://www.noflyzone.org/
19
Where are privacy-related data?
• Direct
– face
– body
– licence plate
• Indirect
– location
– time
– activity
Motivation
• How to protect privacy of individuals captured by
airborne cameras?
• Objective:
– to maintain high fidelity of the visual data
• Solution
– to define & explore the privacy design space
– to configure an adaptive privacy filter
20
Adaptive privacy protection
.
nadir direction camera axis
Design space exploration for adaptive privacy protection in airborne images
O. Sarwar, B. Rinner, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
Adaptive blurring: example
Original OptimalOverFixed Under
Design space exploration for adaptive privacy protection in airborne images
O. Sarwar, B. Rinner, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
21
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
summary
www.eecs.qmul.ac.uk/~andrea
@smartcameras
smartcameras
Autonomous robotic cameras
for collaborative target localization
Acknowledgments
R. Sánchez Matilla, F. Poiesi, T. Nawaz, Y. Wang, B. Rinner, O. Sarwar
EU Artemis JU project COPCAMS
Cognitive & perceptive cameras
EU Erasmus Mundus Joint Doctorate ICE
Interactive & Cognitive Environments
22
References: single camera
Detection of fast incoming objects with a moving camera
F. Poiesi, A. Cavallaro
in Proc. of British Machine Vision Conference (BMVC), York, September 19-22, 2016
Design space exploration for adaptive privacy protection in airborne images
O. Sarwar, B. Rinner, A. Cavallaro
in Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, Aug 23-26, 2016
Trajectory clustering for motion pattern extraction in aerial videos
T. Nawaz, A. Cavallaro, B. Rinner
in Proc. of IEEE Int. Conference on Image Processing (ICIP), Paris, October 27-30, 2014
References: teams of cameras
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
in Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Karlsruhe, August 25-28, 2015
Prioritized target tracking with active collaborative cameras
Y. Wang, A. Cavallaro
in Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
Self-positioning of a team of flying smart cameras
F. Poiesi, A. Cavallaro
in Proc. Int. Conf. on Intelligent Sensors, Sensor Networks and Inf. Proc. (ISSNIP), Singapore, April 2015
Distributed vision-based flying cameras to film a moving target
F. Poiesi, A. Cavallaro
in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept-Oct. 2015