||

Opencast Community SummitUniversity of Cologne, March 14-16

LectureSight in action

Benjamin WulffFraunhofer IAIS

Stephen Marquard and Corné OosthuizenUniversity of Cape Town

|| 02/05/2023 2

LectureSight is an open source real-time video tracking solution for PTZ cameras.

http://lecturesight.org (wiki, code, issue tracker, mailing list)

To run LectureSight, you need:

• a Linux Capture Agent with a GPU• an overview camera (for example a webcam)• a PTZ camera which supports VISCA (Sony), VAPIX (Axis) or ONVIF

About LectureSight

|| 02/05/2023 3

LectureSight:Overview image> detect changes > detect objects> direct camera (track objects)> steer the camera> send commands to PTZ camera (IP / serial)

Capture Agent:Record the PTZ camera image (IP / SDI / HDMI)

How does LectureSight work?

|| 02/05/2023 4

Our installations LS in action… What could possibly go wrong? Production monitoring Future plans

LectureSight at the University of Cape Town

|| 02/05/2023 5

Why? Blackboards, body language, gestures Planned for up to 20 venues Running in production in 4 venues (another 10 pending) Hardware:

PTZ cameras: Axis V5915 (VAPIX), Vaddio ClearView HD-USB (VISCA) Overview cameras: Logitech C930 webcam (USB), Raspberry Pi Cameras (IP) USB active extension cables (up to 20m), ATEN UCE-260 USB 2.0 over CAT5 (up to 60m) Ubuntu 12.04 Capture Agents running Galicaster 1.4.x with Nvidia GPUs (NV310+)

LectureSight scheduling is integrated with the Opencast / Galicaster calendar: tracking starts and stops for scheduled recordings.

LectureSight at UCT: Installations

|| 02/05/2023 6

|| 02/05/2023 7

(Videos)

LectureSight in action

|| 02/05/2023 8

False positives in object detection Lack of object persistence Inability to distinguish presenter from background movement

projection on screen) Inappropriate camera operator strategies (“always follow” not always right)

What could possibly go wrong?

|| 02/05/2023 9

Presenter is tracked, but another object is incorrectly detected, and the camera follows the phantom object for a while.

Falsepositives

|| 02/05/2023 10

When the presenter stops moving, the object times out after a while.

A secondary object can start being tracked instead of the camera remaining on the primary target.

Objectpersistence

|| 02/05/2023 11

Typically too much camera movement from:

rapid movement of a single presentertoo many targets (start/end of lecture)

Rather zoom out (“if in doubt, go wide”) or stop tracking.

Inappropriate camera operator strategies

|| 02/05/2023 12

Has LectureSight started up correctly? Is it tracking, or idle (waiting for an

event)? telnet to your LS instance on port 2501

(osgi.shell.telnet.ip=x.x.x.x) Get status with “lb”, “scr:list”, and

“scheduler:status” Script and graph To do: snapshot / dashboard of overview

camera images

https://opencast.jira.com/projects/LS/issues/LS-153

Production monitoring

|| 02/05/2023 13

University of Osnabrück

University of Manchester

University of Stuttgart

LectureSight is also at …

|| 02/05/2023 14

Improve object tracking (face detection, object persistence, fewer false positives)

More capable camera operator (pan/tilt/zoom, wider range of strategies)

Instrumentation / statistics to give insight into tracking performance and quality

Improved GUI and easier calibration

UCT’s Road Map / Wish List for LectureSight