redefining itu-t p.912 recommendation requirements for subjects of quality assessments in...

7/31/2019 Redefining ITU-T P.912 Recommendation Requirements for Subjects of Quality Assessments in Recognition Tasks

1/23

Introduction

Experimental Method

Results

Conclusion on Standardization

Acknowledgement

Redefining ITU-T P.912 Recommendation

Requirements for Subjects of Quality

Assessments in Recognition Tasks

Mikoaj I. Leszczuk1 Joel Dumke2 Lucjan Janowski1 Artur Kon1

1AGH University of Science and TechnologyDepartment of Telecommunications

2National Telecommunications and Information Administration (NTIA)Institute for Telecommunication Sciences (ITS)

June 1, 2012

1/23
http://find/


2/23

Introduction

Experimental Method

Results


Acknowledgement

Table of contents

1 Introduction

2 Experimental MethodSource Video Sequences

Recognition Test-Plans

3 Results

4 Conclusion on Standardization

2/23
http://find/http://goback/


3/23

Introduction

Experimental Method

Results


Acknowledgement

Introduction

Problems of quality measurements for task-based video partiallyaddressed in Recommendation ITU-T P.912

Subjective Video Quality Assessment Methods for Recognition Tasks

Published: 2008

Introducing:Basic definitions

Methods of testing

Psycho-physical experiments

Section 7.3 (Subjects): Subjects who are experts in the

application field of the target recognition video should be used.

Nevertheless, to best authors knowledge, expert viewer issue not

well verified in specific academic research

Consequently, we compared groups of subjects assessing video

quality for task-based video

3/23


4/23

Introduction

Experimental Method

Results


Acknowledgement

Is Subjects Proficiency Necessary?

Figure: Do I really need to be a security officer in order to participate in a test

checking my ability to read license plate numbers in compressed video?

4/23
http://find/


5/23

Introduction

Experimental Method

Results


Acknowledgement

Source Video Sequences


Table of contents

1 Introduction



3 Results


5/23

I t d ti


6/23

Introduction

Experimental Method

Results


Acknowledgement



Four Categories of Lighting Condition Scenarios

1 Outdoor, daytime light2 Indoor, bright with flashing lights

3 Indoor, dim with flashing lights

4 Indoor, dark with flashing lights

6/23

Introduction


7/23

Introduction

Experimental Method

Results


Acknowledgement



Three Different Distances Used for Clips Creating

1 5.2 meters for indoor scenarios

2 10.9 meters for outdoor scenarios, objects close

3 14.6 meters for outdoor scenarios, objects far

7/23

Introduction
http://find/


8/23

Introduction

Experimental Method

Results


Acknowledgement



Table of contents

1 Introduction



3 Results


8/23

Introduction
http://find/


9/23

Introduction

Experimental Method

Results


Acknowledgement



Viewing Conditions of Room Where Test Took Place

Following ITU-R BT.500-12 and ITU-T P.910

Ratio of luminance of inactive screen to peak luminance: 0.02

Ratio of the luminance of the screen, when displaying only blacklevel in a completely dark room, to that corresponding to peak white:

0.01

Maximum observation angle relative to the normal (this number

applies to CRT displays, whereas the appropriate numbers for other

displays are under study): 30

Ratio of luminance of background behind picture monitor to peak

luminance of picture: 0.15

Other room illumination: low

9/23

Introduction
http://find/


10/23

Introduction

Experimental Method

Results


Acknowledgement



Arranged Viewing Conditions

Figure: A photo of the test environment used in this experiment.

10/23

Introduction
http://find/


11/23

Experimental Method

Results


Acknowledgement



Example of User Interface

Figure: User interface used for subjective target recognition task test performed.

11/23

Introduction


12/23

Experimental Method

Results


Acknowledgement



NTIA Test-Plan (1/2)

NTIA performed the object recognition tests with two groups of

viewers

The Practitioner group

All subjects were volunteers and werent paid for the testMost received invitational travel to Boulder, COAll of them had experience in public safety, including:

Police

Firemen

Emergency Medical Services (EMS)

Forensic Video AnalystsVery few were outside the range of 30-60 years old

Three had minor color vision problems their results were not

significantly different

12/23

Introduction


13/23

Experimental Method

Results


Acknowledgement



NTIA Test-Plan (2/2)

The Non-Practitioner group

Subjects having no experience in image recognitionAll subjects were paid through a temp agency to take the test

None of them had experience in public safety

Subjects had a wide variety of ages, but skewed young

Two had minor color vision problems their results were not

significantly different

13/23

Introduction

E i l M h d


14/23

Experimental Method

Results


Acknowledgement



AGH Test-Plan

Subjects having no experience in image recognition

All subjects volunteers and werent paid for their jobNone of them with experience in public safety area

Almost all subjects 20-25 years old

One of them with color vision problems did no worse than other

viewers so his results included

14/23

Introduction

E i t l M th d


15/23

Experimental Method

Results


Acknowledgement

Introduction to Results

Test results:

For each test: 15540 answers totally givenFor AGH test: 10096 correct and 5444 wrong 65% of right answers

Best and worst conditions

Best recognition outdoor, stationary, close distance scenarios

(89.5% for AGH)

Worst recognition indoor, moving, dark light scenarios (25.4% for

AGH)

15/23

Introduction

Experimental Method


16/23

Experimental Method

Results


Acknowledgement

Results on Recognition Rates

Lighting

Significant influence on recognition, best recognition under daylight

and bright lighting

For 1536 kbit/s bit-rate recognition changes from 92% with bright to30% with dark lighting conditions

Motion

Influence strongly depending on other conditions

For indoor, dim lighting 27% difference between stationary and moving

objects recognition

Distance

Influence larger for lower bit-rates

36% difference between far and close outdoor objects

16/23

IntroductionExperimental Method


17/23

Experimental Method

Results


Acknowledgement

Observations

Subjects accuracy growing

during the test, suggests that

testers learned how objects

were carriedUnder some conditions better

results for CIF resolution videos

(than for higher resolutions)

Often viewers didnt watch

whole clip before giving theanswer

Most of subjects didnt take any

brake during the tests

12

14

16

18

20

22

24

26

28

30

32

0 60 120 180 240 300 360 420

Numberofcorrecta

nswers

Clip number

Figure: Correlation between clip number

and number of correct answers.

17/23

http://find/


18/23

Experimental Method

Results


Acknowledgement

Comparison of Unmotivated with Motivated Subjects

Significant difference between results, about 17% more right

answers in motivated subjects experiments

Under good conditions (high bit-rate, enough lighting etc.) results of

both groups are quite similar, but difference grows fast whileconditions are degrading, than in case of very bad conditions (likefar, moving object) difference decreases again

43% difference for CIF, 64 kbit/s bit-rate, outside, far, stationary objects

clips

6% difference for VGA, 1536 kbit/s bit-rate, outside, close, stationary

objects clips9% differnce for CIF, 64 kbit/s bit-rate, outside, far, moving objects clips

Both, motivated and unmotivated subjects, achieved best and worst

results for same scenarios

18/23



19/23

p a

Results


Acknowledgement


Light changes

For indoor, moving objects scenarios with bright lighting motivated

subjects achieved about 7% higher recognition, for dim lighting

conditions difference grows to 21%, and then fall to 17% for dark

lighting

20

30

40

50

60

70

80

90

100

Bright Dim Dark

PercentofObjectsIdentified[%] Unmotivated_Subjects

Motivated_Subjects

Figure: Moving object scenarios in different lighting conditions.

19/23



20/23

p

Results


Acknowledgement


Motion changes

For inside, stationary, dim lighting conditions scenarios unmotivated

subjects achieved 7% lower recognition, for the same lighting

conditions, but moving objects, difference changes to 21%

50

55

60

65

70

75

80

85

90

Stationary Moving

Percent

ofObjectsIdentified[%] Unmotivated_Subjects

Motivated_Subjects

Figure: Inside, dim lighting, differences between stationary and moving objects.

20/23

http://find/


21/23

Results


Acknowledgement


Distance changes

For close, outdoor, stationary scenarios difference between motivated

and unmotivated subjects recognition is 6%, for far objects motivated

subjects achieved just 1% lower result when unmotivated subjectss

recognition decreased for 24%

50

55

6065

70

75

80

85

90

95

Close Far

PercentofObjectsIdentified[%] Unmotivated_Subjects

Motivated_Subjects

Figure: Outdoor, stationary objects, differences between close and far distances.

21/23



22/23

Results


Acknowledgement


Comparison developed for task-based video

Specifications amendments for ITU-T P.912 Recommendation

developedConsequently first sentence of Section 7.3 (Subjects) of ITU-T

P.912 to get rephrased into: Subjects who are motivated should be

used.

Assisting researchers of task-based video quality to identify subjects

that will allow them to successfully perform psychophysicalexperiment required

22/23


R lt
http://find/


23/23

Results


Acknowledgement

Acknowledgement

European Communitys Seventh Framework Program

(FP7/2007-2013) for received funding for research leading to theseresults under grant agreement. 218086 (INDECT)

U.S. Department of Commerce for received funding for research

under Public Safety Communications Research project

The Department of Homeland Securitys Office of Interoperability

and Compatibility, and NISTs Office of Law Enforcement Standards

23/23
http://find/

redefining itu-t p.912 recommendation requirements for subjects of quality assessments in...

Documents