a guide to picture

8/2/2019 A Guide to Picture

1/20Copyright 1997, Tektronix, Inc. All rights reserved.

A Guide to PictureQuality Measurementsfor Modern

Television Systems


2/20


3/201

Introduction

Convergence is a popular termthese days. It has many defini-tions and many factors dependingon ones perspective and tech-nology base. From a generalistspoint of view, convergence may

be defined as the comingtogether of communication,computer and television tech-

nologies to provide informationof any kind to any location.One of the major focal points ofconvergence is the need for acomplete new technology forthe evaluation of modern televi-sion systems. In this guide, theaspects of video testing are pre-sented based on an understandingof the complete television systemincluding production, compres-sion, decompression and thedisplay or reuse of the originalprogram. The need for continuingapplication of traditional videotesting methods is explainedalong with their limitations forthe identifying the artifactsintroduced by video compression.With the variety of video com-pression methods in use and

being developed, there is arequirement for picture qualityassessment methods which areindependent of the compressionalgorithm and its related artifacts.An overview of subjective testing,which uses a panel of observers,is presented as it has been themainstay for video compressionsystem development. Due to the

complexity and variability ofsubjective testing there is a strongrequirement for an objectivemeasurement instrument muchas we use today for traditionaltelevision systems. Advantagesand limitations of proposedobjective testing algorithms arepresented leading to the conclu-sion that a method based on arepresentation of the humanvisual system is required for

best results. To complete theguide, implementation of an

objective picture quality assess-ment algorithm in a practicalmeasurement instrument isshown to require a combinationof traditional video technologyand modern computer techniques.


4/202

Compressed Television Systems

Compression is Nothing New. Thereare two reasons to compresstelevision video signals, practicallimitations of processing speed(bandwidth) and cost of trans-mission or storage resultingfrom the required bandwidth.Today, the availability of high-

speed semiconductors andintegrated circuits make thelatter reason most important innearly all applications. Virtuallyall video compression methodsutilize the limitations of thehuman visual system toremove the less visible pictureinformation that might otherwise

be present.

As broadcast television wasbeing developed, display ratesof 50 or 60 pictures per secondwere considered necessary. To

provide sufficient visual infor-mation each picture was judgedto need about 500 displayelements (now called pixels) ineach direction.1 To generate andtransmit such a sequence ofpictures in analog form wouldrequire a processing speed andtransmission bandwidth ofabout 10 MHz which was diffi-cult for the available technology

and excessive for the availableradio frequency spectrum.

The first practical televisionbroadcast systems used a formof two-to-one bandwith reduction,or compression, called interlace.Instead of sending 50 or 60frames per second, each frameis divided in to two fields contai-ning half the total number oflines. The lines in the first fieldare every other line from theframe, say lines 1, 3, 5... and thelines in the second field fill inthe missing lines during thesecond field as shown in Figure 1.Picture degradation due tointerlace is in the form of anartifact known as inter-linetwitter, however the quality isquite satisfactory for entertain-ment video viewed severalpicture heights away from thedisplay device.

In the 1950s, color televisionwas developed. A single colorpicture requires three images,specifically red, green, and blue(RGB) for light emitting devicessuch as cathode ray tubes (CRT).Starting from the full progres-sive scan picture, this wouldrequire a 30 MHz bandwidth toprovide the desired picture rate.

Again, interlace is used to reducethe bandwidth to 15 MHz for ananalog RGB system. Within astudio the signals are carriedon three separate cables at5 MHz or more bandwidth each,as shown in Figure 2. A funda-mental compression schemeused in color television is totranslate the three color signals

into the color-differencedomain where the picture isrepresented by a luminance(equivalent to the earlier mono-chrome) picture and two colordifference pictures, R-Y andB-Y. Another name for thissystem is YUV, Y for luminanceand U, V for the two colordifference signals. Again usingthe limitations of the humanvisual system, in this case lesscolor than luminance visualacuity, the bandwidth of the

color difference signals isreduced by 50% for a total YUV

bandwidth requirement of 10MHz. Today, YUV signals areused in both analog and digitalforms and have very little visibledegradation compared to inter-laced RGB video. Both formsare known as component videowith YUV being used formost applications.

Camera LinearMatrix

Red

Blue

Green

Y

R-Y

B-Y

RGB - Component Video - YUV

Quadrature

Modulationat SubcarrierFrequency

Chroma2-5 MHz

Y 0-5 MHz Composite Video

0-5 MHz

R-Y 0-1.5 MHz

B-Y 0-1.5 MHz

Field 1,Odd

NumberedLines

Field 2,Even

NumberedLines

Camera VideoSignal

Figure 1. Interlaced scanning. Figure 2. Analog television compression.

1The Americas, Japan and Korea use the 525-line, 60 field/s system while most of the rest of the world uses the 625-line 50 field/s system.


5/203

Color television in the 1950s,and until recently, required fur-ther compression both to fit inthe allocated 6 MHz bandwidthof transmission channels and to

be compatible with the installedbase of monochrome televisionsets. To accomplish this task thetwo color difference signals arefurther reduced in bandwidth to

about 1.5 MHz each and quadra-ture modulated on a subcarrierwhich is subsequently added tothe luminance signal producingcomposite video. CompositeNTSC and PAL2 produce verygood entertainment qualitycolor video in a 5 MHz band-width, a compression ratio ofsix to one from the ideal pro-gressive scan RGB video. Thefinal two to one compressionfrom component to compositedoes bring with it noticeable

picture degradation includingchroma information seen as in-correct luminance and visa versa.3

Today, using modern digitalcompression methods, four ormore excellent quality digitalcomponent television signalscan be delivered to the homewithin the same 6 MHz trans-mission channel. If derived froma high quality digital component

measurement instrument to pro-vide a result that is independentof the compression method used.

The compression method that isbecoming dominant today iscalled MPEG-2, defined by theMotion Picture Experts Groupand standardized by both theInternational StandardsOrganization (ISO) and the

International ElectotechnicalCommission (IEC). MPEG-2 is

based on the Discrete CosineTransform (DCT) method incombination with powerfultemporal compression tech-niques.5 Although some applica-tions may be best served byother compression methods,MPEG-2 is expected to be themost widely used method in theforeseeable future. This is

because it is an agreed standardthat is either optimum or goodenough for a wide variety ofapplications, a large amount ofeffort is going into the develop-ment of chip sets for lower costencoders and decoders, and theforthcoming large installed basewill be attractive for manyequipment manufacturers andapplication developers.

source, the resulting multipletelevision signals have a notice-able quality improvement overthe single 6 MHz bandwidthcomposite video signal.

Digital Compression Methods.Digital video became a reality in1973 with the invention of thecomposite-based digital time

base corrector for video taperecorders. In the early 1980s, aworldwide digital componentvideo standard was developedrequiring 216 Mb/s or 270 Mb/sdepending on the use of 8-bitor 10-bit sample values. Thisstandard is commonly known asRec. 6014. It is the dominantsampling structure for digitaltelevision and its use is showingrapid growth for all types ofapplications. Since the approvalof the Rec. 601 standard, much

research and development hasbeen directed towards digitalvideo data rate reduction resultingin a variety of video compressionmethods. Each of these com-pression methods has its ownadvantages, disadvantages andpicture degradation characteris-tics. It will be important for anygeneral purpose picture quality

2NTSC (National Television System Committee) is used in most 525-line countries and PAL (Phase Alternating Line) is used in many, but not all, 625-line countries.3For a more complete description of basic digital t elevision see "A Guide to Digital Television Systems and Measurements", Tektronix literature number 25W-7203.4Rec. ITU-R BT.601, Encoding Parameters of Digital Television for Studios. Originally it was CCIR Recommendation 601 but has been changed to Recommendation ITU-R BT.601. Rec 601 is used throughout

this guide (and is much easier to say).5See "MPEG-2 Fundamentals for Broadcast and Post-production Engineers" for a brief description of the MPEG compression method. Tektronix literature number 2AW-1061.


6/20

The Modern Television System.

A simplified block diagram of amodern compressed televisionprocessing and transmissionsystem is shown in Figure 3.Television nominally consists ofaudio and video; however, thesystem may include data andcontrol signals (not shown inthe figure) hence may be

thought of as a multimediasystem. One-direction transmis-sion is shown. A multiplicity ofmethods are depicted, particu-larly in transmission, makingthis diagram an overview ofmany types of applications. Nospecific compression method isshown, however the MPEG-2transport stream is shown in thetransmission area since it can beconsidered a general purposemultiplexing scheme capable ofcarrying any type of compressed

video and audio.Analog RGB video is producedin the camera and processedinto one or more of severalpossible formats; analog com-posite, digital composite, analogcomponent or digital compo-nent. Full-bandwidth digital

video is an extremely importantpart of the television systemtoday. Program productionprocesses must be full-bandwidthdigital (or analog) in order tomanipulate the picture toproduce desired artistic results.

Following program production,the television signal may becompressed for storage, efficienttransmission or intra-facilityinterconnection in digital form.Typically this will be MPEG-2compression resulting in anMPEG transport stream (MTS)that may be multiplexed withother MPEG transport streams fortransmission or interconnection.

New systems for RF transmis-sion of television signals usedigital modulation schemeswhich are generally more robustfor the same transmitted power

and provides the digital channelfor compressed television signals.It is important to note, that evencompressed digital video broad-casting to the home often startswith full bandwidth digitalvideo to drive the bit-rateefficient, statistically-multiplexedcompression system.

The broadband telecommunica-tion system provides a varietyof transmission methods.Traditionally these have beenvoice channel oriented withspecial data mapping for digitaltelevision signals. Althoughdirect mapping of the MTS intothe digital telecommunicationshierarchy is in the process of

being standardized, it is expect-ed that asynchronous transfermode (ATM) will become thepreferred method of inter-facili-ty video transfer.

Video testing in this moderntelevision system is not just amatter of developing new tech-niques to evaluate the effects ofcompression. The significantportion of the system utilizinganalog and full-bandwidth digitalsignals requires application oftraditional analog and recentlydeveloped digital test methods.To determine picture qualityimpairments caused by com-pression, a video measurementsystem must take into accountthe various signal formatchanges affecting the videothroughout the system.

4

Camera CCU CompressTransport

Multiplexer

MTS

V-RGB

V

AAudio

V

A

MTS

OtherPrograms

MTS (Point-to-Point or Network)

RFDemod

RFModulator

ATMGateway

ATMGateway

Satellite, Cable,Terrestrial Transmission

BroadbandNetwork

QPSK

QAM

TDM

QPSK

QAM

TDM

TransportDemux

DecompressMTS V

A

Distribution QualityTVSet

PRODUCTIONRecording

EditingSpecial Effects

PRODUCTIONRecording

Editing

Special Effects

Contribution Quality

V

A

MTS

OtherPrograms

Figure 3. Modern television system.


7/20

Figure 4. Analog video system.

Video Testing Concepts

Growth of Functional Layers. Overthe half-century of widespreadtelevision use there has been arelatively simple model foranalyzing analog video systems.Figure 4 shows a basic blockdiagram of the analog videosystem, its functional layers and

test methods. Testing is per-formed at one interconnectiongenerally carrying a compositePAL or NTSC signal. A singlemeasurement instrument cananalyze both the operationalaspects, such as signal level orcolor balance, and the data for-matting which is the synchro-nizing signal part of the samevideo waveform. This analysisof the signal quality through thetransmission path using a suiteof test signals does an adequate

job of characterizing resultingpicture quality. The idea of asuite of test signals is important.No one test signal will charac-terize the system and someexpert interpretation as well asvisual inspection of the resultingpictures is required. For intra-facility transmission of signalson coaxial cable, a separatepiece of test equipment, thetime domain reflectometer, isused to ensure the continuity ofthe physical layer. Long range

transmission is by amplitude orfrequency modulation on acarrier, however the resultingchannel characteristics for thevideo are still determined byanalog measurements such asthose specified in ANSI T1.502.6

5

6ANSI Standard T1.502 "System M-NTSC Television Signals Network Interface Specifications and Performance Parameters."

CameraComposite

Encoder

RGBPAL/NTSC Transmission

orOperation

Recordor

Display

PAL/NTSC

VIDEOOperational Monitoring

Technical Measurements

SYNCHRONIZING WAVEFORMSTechnical Measurements

PHYSICAL LAYER (Coax)Technical Measurements

Functional Layers

Waveform Monitorsand

Measurement Sets

TDRs

RGB

CameraPAL/NTSC Transmission

orOperation

Recordor

Display

PAL/NTSC

VIDEO

SIGNAL CODING (Rec 601)

DIGITAL FORMATTING (Rec 656)

Functional Layers

Waveform Monitorsand

Measurement Setswith

Analog or DigitalCapabilities

TDRs, OTDRs

StudioInterconnector Operation

EncodeD/A

Rec 601/656 Rec 601/656

DIGITAL WAVEFORM (Rec 656)

PHYSICAL LAYER (Coax/Fiber)

Operational MonitoringTechnical Measurements

Technical Measurements

CompositeEncode orRec 60 1/656

Format

DecodeA/D

Figure 5. Hybrid digital/analog video system.


8/20

With the advent of digital televi-sion over the past 15 years, amore complex system blockdiagram and set of functionallayers has been required asshown in Figure 5. The analogsignal is converted to digital inaccordance with a samplingstandard such as Rec. 601.Formatting and studio intercon-

nection of the digitized signalfollow a related standard, Rec.656,7 leading to a extension inthe functional layers and thevariety of tests to be performed.For operational purposes, themonitoring of analog videosignal properties is still key;however, this signal must beprocessed from the digital data.

Where testing of the analogsignal required only that variousparameters be measured on asingle waveform, digital testingrequires analysis of the digitalwaveform, digital data formattingand digital signal coding inaddition to the resulting analogsignal. Again a suite of testsignals is required, expanding

the suite needed for analog-onlytesting. Although all those mea-surements can be performedwith a signal instrument, suchas the Tektronix WFM601M,there is significant processing

between each pair of layers withdifferent analysis methods foreach layer as well. Prior to theadvent of digital compression

techniques, transmission of thishigher quality signal was handled

by compression back to thecomposite analog domain. Theanalog-to-digital and digital-to-analog conversion does introducesome signal quality degradation

beyond that of the basic NTSCor PAL analog signal.

With the convergence of televi-sion and telecommunication,not only are there many morefunctional layers for the testengineers to consider but thereare various possible paths withdifferent layers. Figure 6 showsa few of the possible functionalpaths and layers.

Serial Digital Interconnect (SDI)is the Rec. 656 worldwidestandard used for serial digitalvideo. The SMPTE WorkingGroup on Packetized Television

Interconnections is developing amethod of carrying packetizeddata over the same cabling andswitching hardware calledSDDI (for Serial Digital DataInterconnect). A networkingtype interconnect for the televi-sion facility, being considered

by ANSI and SMPTE, is FibreChannel which provides highspeed, large packet sizes andreasonably priced hardware. TheSynchronous Digital Hierarchy(SDH) telecom methods are well

established worldwide and candirectly carry the MPEG-2 trans-port stream with simple dataformatting, although there ispresently no standard. Lookingtoward the future, ATM is theexpected method for transmis-sion of packetized data, certainlyfor long distances and perhapswithin a studio.

6

Data,etc.

MPEGAudio

Encoding

MPEGVideo

Encoding

TELEVISION SIGNALSVideo, Audio and Related Test Signals

Analog to Digital Conversion

MPEG System (Transport Stream)

SignalDataFormating

OtherAudio

Encoding

OtherVideo

Encoding

DVB, GA,Other

Call

Setup,SwitchingControl

Transmitter and Receiver Electro/Optic Circuits

Copper or Fiber and Connectors

SwitchingHardware

Fibre ChannelSDI

SDDI

SDH/Sonet

ATM

AAL(1or 5)

PhysicalLayer

TypicallyHardware

TransmissionData

Formatting

Figure 6. Modern television functional layers.

7Rec. ITU-R BT.656, "Interfaces for Digital Component Video Signals in 525-line and 625-line Television Systems Operating at the 4:2:2 level of Recommendation 601."


9/20

Three key testing layers can bedefined for the modern televi-sion system as shown in Figure 7.Each has its own subset of moredetailed testing layers. Videoquality for compressed televi-sion systems is a much morecomplex matter than just usingthe indirect measurementmethods for uncompressed

video. This will be covered indetail in subsequent sections ofthis guide. Once the picture has

been compressed, the resultingdata is formatted for intra-facilityconnections. Examples for theuse of such connections are:program interchange betweenvideo disk servers or severalvideo/audio encoders sendingsingle program transportstreams to a multiplexer to pro-duce a multi-program transportstream for satellite broadcasting.

This is an appropriate layer forprotocol testing because the dataformatting can be quite complexand is relatively independent ofthe nature of the uncompressedsignals or the eventual conversionto inter-facility transmissionformats. For a majority of thetelevision transmission systemsthe MPEG-2 transport stream isthe common denominator at thecompressed data level. Thesyntax and semantics for boththe compressed data and the

transport stream are welldefined. Typical protocol testingequipment, such as theTektronix MTS 100, will be botha source of known valid, orspecifically invalid, signals andan analyzer which locates errorswith respect to a defined standardand determines the value of var-ious operational parameters forthe stream of data. There are anumber of possible inter-facilitytransmission methods as previ-ously described. Many are well

established, such as SDH/Sonetand cable television, with avariety of effective test equip-ment available. ATM is anemerging technology with newtest equipment on the market

and under development.Adaptation of traditional com-munication test equipment toanalyze or interconnect withMPEG-2 transport streams is onthe horizon.

Video Quality. There are severaldimensions of video qualitymeasurement methods that needdefinition. These are summa-rized in the table below.

Subjective measurements arethe result of human observersproviding their opinion of thevideo quality. Objective mea-surements are performed withthe aid of instrumentation,manually with humans readinga calibrated scale or automaticallyusing a mathematical algorithm.

Direct measurements areperformed on the material ofinterest, in this case, picturesand are also called picturequality measurements.

Indirect measurements aremade processing speciallydesigned test signals in thesame manner as the picturesand are also called signalquality measurements.

Subjective measurements areonly done in a direct mannersince the human opinion of testsignal picture quality is not par-ticularly meaningful. (Of course,expert viewing of full-field testsignal pictures is useful as a wayto determine signal distortionsnot for their aesthetic value.)

In-service measurements aremade while the program is

being displayed, directly by

evaluating the program materialor indirectly by including testsignals with the program material.Out-of-service, appropriate testscenes are used for directmeasurements and full-fieldtest signals are used forindirect measurements.

7

SIGNAL TEST FUNCTION

NTSC/PALComponent(Rec. 601)

MPEG-2Transport Stream

Telecommunicationor RF Channel

Video Quality

Protocol Analysis

(Origination Side Shown,Destination Side Similar)

TransmissionSystem

Video Input

MPEG Compressionand

Transport StreamFormatting

ConnectionsINTRA-Facility

INTER-FacilityConnections

ATM Output RF Output

ChannelFormattinge.g. OC-3

SwitchFormattinge.g. ATM

Cable orSatelliteChannelCoding

andModulator

Figure 7. Functional testing layers.

Subjective Direct

(Picture Quality)

Objective Direct(Picture Qualilty)

Objective Indirect(Signal Quality)

In-Service Out-of-Service

Program

Material

ProgramMaterial

Vertical IntervalTest Signals

Test

Scenes

TestScenes

Full FieldTest Signals


10/20

Although there is a modestamount of compression appliedto the NTSC and PAL compositesystems, they are considereduncompressed in today's termi-nology. Signal quality (objectiveindirect) measurements are areasonably good way to deter-mine the picture quality forsuch uncompressed systems.

That is, there is a strong mathe-matical correlation betweensubjective measurements madeon pictures from the system andobjective measurements madeon a suite of test signals usingthe same system. The correlationis not perfect for all tests. Thereare distortions in compositesystems, such as false colorsignals caused by high frequencyluminance, which are not easilymeasured by objective means.Also, there are objective mea-

surements which are so sensitivethey dont directly relate tosubjective results. However,such objective results are oftenvery useful because their effectwill be seen by a human observerif the pictures are processed inthe same way a number oftimes. An example would bemultiple generations using ananalog video tape recorder.

The reason signal quality mea-surements work with analog and

full-bandwidth digital systemsis uncompressed systems arelinear.8 That is, the system

behavior is time invariant,signal independent and super-position applies. Signal qualitymeasurements are made with asuite of test signals whoseresulting distortions willdetermine transmission channelor video processing characteris-tics. These test signals can bevery short, as an example, oneline in the vertical interval. Signal

quality of the uncompressed

video remains critical in systemsthat use compression forseveral reasons:

The input to a video compres-sion codec must be accurate,in compliance with approp-riate standards, and of as higha quality as possible toprovide for efficient encoding.

Video processing such asadding titles and specialeffects can not be accomplishedin the compressed domain.

Production facilities will notbe fully compressed due tothe cost and quality ofcompression codecs.

The only way for differentcompressed formats to beinterchanged is at the full

bit-rate level.

This leads to a strong require-

ment for testing of the analogand full bandwidth digitalportions as well as the sophisti-cated compression andtransmission systems.

With the advent of compresseddigital video systems the situa-tion has become more complex.Signal quality testing will notwork for the compressionencoder/decoder part of the sys-tem. Traditional test signals arerelatively simple compared to anatural scene and are easilycompressed with little distor-tion or loss. Due to the ease ofcompression, these signals donot evaluate the encoder/decoderprocess. As an example, signal-to-noise ratio is not a reliablemeasure of picture quality, it isnot a constant for a given systemand it can give completelymisleading results. Thereforepicture quality measurementsrequire a direct method, usingnatural scenes, or an equivalentthereof, which are much more

complex than traditional testsignals. These complex scenesstress the capabilities of theencoder resulting in non-lineardistortions that are a function ofthe picture content.

Use of digital compression hasexpanded the types of distor-tions that can occur in themodern television system.

Quantization noise which isalso present in full-bandwidthdigital systems is oftenincreased by the compressionsystem bit rate reductionprocess. Blockiness is a checker-

board pattern that may occur inDCT-type compression systems.Loss of resolution is common

because the compressionsystems use the human visualsystem limits of acuity as aguide for removing informationfrom the picture. Therefore,greater compression generallymeans less resolution. Althoughhuman acuity is less for chroma,the uncompressed picture hasalready used some of that lati-tude and compression systemsoften squeeze the chroma evenmore than the luminance. Edge

busyness is another effect ofquantization since more infor-mation is removed from thehigh-resolution parts of the pic-ture producing noise on edges.When that noise is displaced bythe compression processing intonearby flat areas it is sometimescalled mosquito noise. Motionrelated artifacts, such as jerki-ness or misplaced blocks ofpixels, are present in systemswhich use temporal compres-sion either based on sophisticat-ed motion compensation orsimply dropping frames becausethere are not enough bits avail-able in low bandwidth systems.9

8

8Analog systems are not perfectly linear, however they are quite good and sensitive objective testing can be used to determine the small amounts of non-linearity.9A list of impairment terms and other definitions may be found in ANSI T1.801.02 Digital Transport of Video Teleconferencing, Video Telephony Signals - Performance Terms, Definitions and


11/20

With the broader range of distor-tions to measure and the desireto optimize program distribu-tion both technically and eco-nomically, the field of subjectivemeasurement has expanded.Some of the subjective measure-ments even include an elementof program quality as well aspicture quality as will be

discussed in detail later in thisguide. Since signal qualitymeasurements will not do thejob, objective picture qualitymeasurements are needed.Expanded types of signal qualitymeasurements are not appropri-ate to cover the new subjectivemethods. In fact, with theincreased ideas for subjectiveevaluation it may be true thatthe traditional signal qualitymeasurements no longer have asstrong a correlation with subjec-

tive requirements. There doesnot appear to be any plan toexpand or re-test the signalquality measurement methodssince there is so much work todo in developing objective pic-ture quality methods. Such pic-ture quality measurement meth-ods must, also, have strong cor-relation with subjective mea-surements and cover a reason-ably broad range of subjectiveconsiderations. It is expectedthat picture quality distortions

too small for the human to seewill be measured and providean indication of the performanceof concatenated systems.

Picture Quality Testing

Subjective Testing. Televisionprograms are produced for theenjoyment or education ofhuman viewers so it is theiropinion of the video qualitywhich is important. Informaland formal subjective measure-

ments have always been, andwill continue be, used to evaluate

system performance from thedesign lab to the operationalenvironment. Even with all theexcellent objective testing meth-ods available today for analogand full-bandwidth digitalvideo, it is important to havehuman observation of thepictures. There are impairmentswhich are not easily measured

yet are obvious to a humanobserver. This situation certainlyhas not changed with the addi-tion of modern digital compres-sion. Therefore, casual or infor-mal subjective testing by areasonably expert viewerremains an important part ofsystem evaluation or monitoring.

Formal subjective testing hasbeen used for many years with arelatively stable set of standardmethods until the advent ofdigital compression subjectivetesting described in Rec. 500.10

In brief, a number of non-expertobservers are selected, tested fortheir visual capabilities, showna series of test scenes for about10 to 30 minutes in a controlledenvironment and asked to scorethe quality of the scenes in oneof a variety of manners.Subjective testing is used for

both quality assessment, systemperformance under optimumconditions, and impairmentassessment under non-optimumperformance due to transmis-sion limitations. In a moderntelevision system that incorpo-rates compression, the picturequality is not a constant overtime. Picture quality is a func-tion of the complexity of theprogram material and, in thecase of statistical multiplexing,the moment to moment operationof the transmission system.Considering this time varyingproperty and the number of new

impairments, the defined andproposed measurement methods

have grown in recent years. Inaddition to selection of themeasurement method there are anumber of other proceduralelements for which alternateapproaches are available. Theseare such things as: viewingconditions, choice of observers,scaling method to score theopinions, reference conditions,

signal sources for the testscenes, timing of the presenta-tion of the various test scenes,selection of a range of testscenes, and analysis of theresulting scores. Selection of theparameters for each of theseelements is related to theintended application of thetelevision system and leads to acomplex maze of possibilities.A description of the varioussubjective measurement methodsprovides some insight.

Double Stimulus ImpairmentScale (DSIS) Observers areshown multiple reference-scene, degraded scene pairs.The reference scene is alwaysfirst. Scoring is on an overallimpression scale of impair-ment: imperceptible, percepti-

ble but not annoying, slightlyannoying, annoying, and veryannoying. This scale is com-monly known as the 5-pointscale with 5 being imperceptibleand 1 being very annoying.

Double Stimulus ContinuousQuality Scale (DSCQS) Observers are shown multiplescene pairs with the referenceand degraded scenes randomlyfirst. Scoring is on a continuousquality scale from excellent to

bad where each scene of thepair is separately rated but inreference to the other scene inthe pair. Analysis is based onthe difference in rating foreach pair rather than the

absolute values.

9

10The standard for subjective measurements is ITU-R BT.500 "Methodology for the Subjective Assessment of the Quality of Television Picture". First issued in 1974 and formally known as CCIR Rec. 500, version 7

of this document covers all of the past and proposed methods for subjective testing.


12/20

Single Stimulus Methods Multiple separate scenes areshown. There are twoapproaches: SS with no repe-tition of test scenes and SSMRwhere the test scenes arerepeated multiple times.Three different scoring methodsare used:

Adjectival the 5-gradeimpairment scale, howeverhalf-grades may be allowed.

Numerical an 11-gradenumerical scale, useful ifa reference is not available.

Non-categorical acontinuous scale with nonumbers or a large range,e.g. 0 - 100.

Stimulus Comparison Method Usually accomplished withtwo well matched monitors

but may be done with one. Thedifferences between scene pairsare scored in one of two ways:

Adjectival a 7-grade,+3 to -3 scale labeled: much

better, better, slightly better,the same, slightly worse,worse, and much worse.

Non-categorical acontinuous scale with nonumbers or a relation-number either in absoluteterms or related to astandard pair.

Single Stimulus ContinuousQuality Evaluation (SSCQE) A program, as opposed toseparate test scenes, is contin-uously evaluated over a longperiod, 10 to 20 minutes. Datais taken from a continuousscale every few seconds.Scoring is a distribution of theamount of time a particularscore is given. This methodrelates well to the time variantqualities of todays compressed

systems, however it tends tohave a significant content ofprogram quality in addition tothe picture quality.

In one evaluation, Rec. 601video, which has been consid-ered to be essentially perfect forthe past fifteen years, was givena quality rating above 90% foronly 14 minutes out of a 20minute program.

In addition to these definedmethods, there are two newapproaches that start to bridge

the gap between subjective andobjective picture quality mea-surements. They are picture-content failure characteristicsand composite failure charac-teristics of program and trans-mission conditions. These will

be discussed in the section onobjective measurements.

Advantages of subjective testingare;

valid results are produced forboth conventional and com-

pressed television systems,

a scalar mean opinion score(MOS) is obtained, and itworks well over a wide rangeof still and motion pictureapplications.

Weaknesses of subjective testingare;

a wide variety of possiblemethods and test elementparameters must be considered,

meticulous setup and control

are required, many observers must be

selected and screened, and the complexity makes it

very time consuming.

The net result is subjective testsare only applicable for develop-ment purposes. They do notlend themselves to operationalmonitoring, production linetesting or trouble shooting.

Objective Testing. The need for anobjective testing method of pic-ture quality is clear; subjectivetesting is too complex andprovides too much variability inresults. However, since it is theobservers opinion of picture

quality that counts, any objec-tive measurement system musthave good correlation withsubjective results for the samevideo system and test scenes. Aswith subjective testing, nearlyall objective testing methods donot claim to measure picturequality directly but provide anindication of how a degraded

picture or scene compares witha reference picture or scene.Such comparisons tend toeliminate the aspect of programquality from the measurements.

Over the past few years a widevariety of methods have beeninvestigated for objective testingof picture quality in compressedvideo systems. The methodsproposed may be roughlydivided into two categories,feature extraction and picturedifferencing, each of which may

be implemented in a varietyof ways.

Feature extraction uses amathematical computation toderive characteristics of a singlepicture (spatial features) or asequence of pictures (temporalfeatures). This usually resultsin an amount of data per picture(say, a few hundred bytes) thatis considerably less than usedto transmit the compressedpicture. The calculated char-

acteristics of the reference anddegraded pictures are thencompared to determine anobjective quality score.

Picture differencing uses amatrix-based mathematicalcomputation to process eachpicture or sequence of pictures.The resulting data representsa filtered version of the picturescontaining an amount of datasimilar to the original pictures.Usually, the pixel-by-pixeldifference between filtered

versions of the reference anddegraded pictures is used todetermine an objective qualityscore. In some cases, it may bethe difference between thereference and degraded picturesthat is filtered.

10


13/20

Figure 8 shows how the twobasic methods might be used inan objective measurement sys-tem. The advantage of the fea-ture extraction method (8a) isthe calculated characteristics ofthe reference (input) picturemay be sent through the trans-mission channel along with thecompressed picture for objective

scoring at a remote location.Because of this advantage, thefeature extraction method has

been vigorously pursued, some-times in combination with thepicture differencing method.However, research at Tektronixand other laboratories has shownthat certain picture differencingmethods (8b) provide objectivescores that correlate best withsubjective results.

It is important to note, neitherof these methods can be guaran-teed to always give the correctpolarity of the change in picturesalthough virtually all systemsproduce picture degradation.There are examples where apicture with noise or otherartifacts is improved by filtersat the input to a compressionsystem resulting in a net pictureimprovement through the com-pression/decompression process.

Some of the concepts of thefeature extraction method are

codified, for luminance only, ina recently approved AmericanNational Standards Institute(ANSI) standard.11 The standardmay be considered a tool box ofobjective measurement methodsproviding a set of performanceparameters where each parameteror combination of parameters issensitive to some uniquedimension of video quality orimpairment type. The scopeof the standard statesDiscrimination between two or

more similar systems is beyondthe accuracy of the objective

measurements defined in thisstandard at this time. Furtherwork by the members of theANSI committee has beenreported12 indicating that a com-

bination of feature extractionand picture differencingmethods give the best results.Even with these extensions, themethods to be used should be

chosen depending on the appli-cation to provide the bestcorrelation between subjectiveand objective scores.

Another significant approach tofeature extraction has beendeveloped and reported in thelatest proposed revision to theinternational subjective testingstandard, Rec. 500, as appen-dices Picture-content failurecharacteristics and Compositefailure characteristics of programand transmission conditions.They introduce the concept of

criticality which is a measureof the complexity of the picturesto be compressed. The idea isthat pictures with more criticality(complexity) will be more diffi-cult to compress and will resultin lower picture quality.

This approach is compressionmethod dependent as well asapplication dependent. Differentcompression methods willproduce a different picturequality for the same input criti-cality. Even the same method,for instance MPEG-2, willproduce different results ifparameters are changed such asgroup of picture length or rela-tive number of bits allowed forluminance verses chrominance.The method of calculatingcriticality will be dependent onapplication much as the featureextraction methods are whenapplying the ANSI techniques.

11

11ANSI Standard T1.801.03-1996 "Digital Transport of One-way Video Signals, Parameters for Objective Performance Assessment".12ANSI T1A1.5/96-121 "Objective and Subjective Measures of MPEG Video Quality".

ReferencePicture

Compression Codec DegradedPicture

FeatureExtraction

FeatureExtraction

FeatureComparison

Objective

Score

Low Bandwith Data

(a Few 100 Bytes)

Figure 8a. Feature Extraction

ReferencePicture

Compression Codec DegradedPicture

ImageProcessing

ImageProcessing

PictureData

DifferenceObjective

Score

Picture Data

(Many kBytes)

Picture Data

(Many kBytes)

Figure 8b. Picture Differencing


14/20

is worse with a value of 27.10.Therefore, MSE is not an appro-priate picture differencinmethod for objective picturequality measurements.

Although picture differencingmethods based on featureextraction parameter calculationshas been shown to improve onthe basic ANSI approach, theresult is not application or tech-nology independent. Numerousresearchers have indicated thatthe way to achieve technologyindependence and provide goodcorrelation between subjectiveand objective measurements isto have the test instrumentperceive and measure videoimpairments in the same manneras a human observer. In otherwords, filtering for the picturedifferencing method should usea model of the human visualsystem (HVS). Application ofsuch a model will provide animage quality metric that isindependent of video material,types of impairments and thecompression system used.

As previously stated, certainpicture differencing methodsprovide better objective picturequality measurement correlationwith subjective results. Themost obvious picture differencingmethod is to simply subtract thetwo pictures without any filteringor processing. If the differenceis zero, the pictures are identical.

When the pictures are differenta mean square error (MSE) can

be calculated on a pixel by pixelbasis, a larger MSE indicates agreater difference betweenreference and degraded pictures.Another way to express thisdirect picture difference isPSNR which computes the logof the ratio of the square of thepeak signal (255hex in an 8-bitsystem) to the MSE much as isdone for signal to noise ratio(SNR) in an analog system. This

method has some practical usesand some significant failings.For a very constrained system,say bit rate change only, MSEwill increase with decreasingpicture quality. Also, designers

may find it useful to view thepixel value differences in pictureform when looking for designproblems. However, it is wellknown that MSE can give acompletely false indication. Asan example, consider the com-parison of two types of degrada-tion; one is the addition of asmall amount of random noise,

say five quantizing levels, andthe second is the addition ofsomewhat less blockiness, saytwo quantizing levels. The latterimpairment will have a smallerMSE value, however observerswill consider the noisy pictureto have little degradation wherethe blockiness will be quiteapparent as a significant degra-dation. An example of this mea-surement is shown in Figure 9.Codec A provided an outputimage with a MSE value of

21.26 but a significant amountof blockiness whereas codec Bprovides a much better lookingpicture with a small amount ofadded noise, however the MSE

12

Input Image Codec A: MSE = 21.26 Codec B: MSE = 27.10

Figure 9. MSE measurement examples.


15/20

Application of the Human VisualSystem. Researchers at the DavidSarnoff Research Center (SarnoffLabs) have devoted significantresources, over a number ofyears, to studying the humanvisual system and applying theknowledge gained to televisiondisplay and picture quality eval-uation. Based on this work, the

Just Noticeable Difference (JND)image quality metric has beendeveloped for automatically andaccurately assessing the percep-tual magnitude of differences

between a test and referencesequence.13 Figure 10 shows anoverview of the JND modelarchitecture. The inputs are twosequences of arbitrary lengthwhich are separately processed(filtered) to the DifferenceMetric box near the bottom ofthe diagram where the differ-

ences between the processedsequences are used to developthe JND maps and JND numericvalues. An example is shown inFigure 11. Image A is the refer-ence and image B is the degrad-ed picture, image C is the JNDmap. Note the distortion of thenumbers on the trolley car andthe corresponding bright area inthe JND map. Also note thesolid line on the ground to theleft of the trolley car which has

become a dotted line in the

degraded picture. In the JNDmap, a series of dots shows thenoticeable difference betweenthe two pictures.

For the JND image quality metriccalculation, each field of thesequence is represented as a trioof RGB images. In the first stage,labeled Front End Processing,

the voltage units are transformedto light output units to obtainluminance (Y), and then to thepsychophysically defined quan-tities of the CIE L*u*v* uniformcolor space to obtain the twochannels (u*, v*) of the modelschrominance pathway. In the nextstage of the model, labeled PyramidDecomposition, each sequence

is filtered and down-sampledusing a Gaussian pyramid

operation to efficiently generatea range of spatial resolutions forsubsequent filtering operations.Next, the Normalization stagesets the overall gain with a time-dependent average luminance, tomodel the visual systems rela-tive insensitivity to overall lightlevel and to represent sucheffects as the loss of visual sen-

sitivity after a transition from abright to a dark scene.

13Material for this section of the guide is excerpted from the paper Vision Model-based Assessment of Distortion Magnitudes in Digital Video by J. Lubin, M. Brill and R. Crane, presented at the Made to Measure

96 symposium, Montreux, Switzerland, November 1996.

R' G' B'

Front End Processing

Pyramid Decomposition

Y u* v*

Normalization

Oriented Contrast Flicker Contrast Chromatic Contrast

Contrast Energy Masking

JNDValues

DataPooling

ChromaJND Map

LumaJND Map

DifferenceMetric

Identical Process

R' G' B'

Figure 10. JND image quality metric architecture.

A B C

Figure 11. JND map example.

13


16/20

After normalization, threeseparate contrast measures arecalculated; oriented, flicker andchromatic. In each case, thecontrast is a local difference ofpixel values divided by a localsum, approximately scaled as afunction of pyramid level so theresult is 1 when the image con-trast is at the human threshold.

This establishes the definitionof 1 JND, which is passed tosubsequent stages of the model.(The JND unit of measure isfunctionally defined such that 1

JND corresponds to a 75% prob-ability than an observer viewingtwo images multiple times would

be able to see the difference.)

In the Contrast Energy Maskingstage, each contrast image issubjected to a point non-linearity,the gain of which is controlled

by the response across otherresolutions and channels. Thisgain-setting is included tomodel visual masking effectssuch as the decrease in sensitivityto distortions in busy image

regions. The parameters of thepoint non-linearity at this stageare fit according to contrastdiscrimination data in whichthe contrast increment neededto detect the change in contrastis measured as a function of thecontrast from which the changeis made.

At the Difference Metric stage,outputs from the test and refer-ence sequences are combinedvia a simple difference operatorand then summed across pyramidlevels and channels to returnthe number of JNDs in bothluma and chroma. Separate JNDmaps for luma and chroma can

be pooled into one map andsummary statistics can beobtained. Such statistics would

be mean JND, max JND andQ-norm, which allows a general-ized approach to mean andmax calculations.

The JND image quality metricprovides all the facilitiesrequired for a robust objective

picture quality measurementmethod. It includes the threenecessary dimensions for evalu-ation of dynamic and complexmotion sequences; spatial analy-sis, temporal analysis and fullcolor analysis. By using a modelof the human visual system in apicture differencing process,results will be independent of

the compression process andresulting artifacts. This is partic-ularly important in concatenat-ed television systems which areexpected to involve several dif-ferent compression methods.14

Objective measurement methodsthat rely on a model of the com-pression codec or evaluate spe-cific types of artifacts will havevery limited application in suchsystems. In addition to beingappropriate for overall systemmeasurement, it is expected that

combining the results of theJND image quality metric forseparate parts of a concatenatedsystem will provide a usefulindication of overall performance.

14

14An example of up to ten different compression methods in a complete television system is described in the paper "Why is Objective Evaluation Needed for Compressed Digital Video", by C. Dalton, presented at the

Made to Measure '96 Symposium, Montreux, Switzerland, November, 1996.


17/20

System Approach to ObjectiveTesting. Objective testingrequires a valid algorithm, suchas the JND image quality metric,as its foundation. However,implementation of a real-worldmeasurement system mustinclude a number of otheraspects such as: reference scenemotion sequences, a physical

source for the reference scenes,format conversions, scenechanges due to processing in thenon-compressed parts of thesystem, and accurate alignmentof pictures as an input to themeasurement algorithm. Anoverall block diagram of themeasurement system is shownin Figure 12 for application ofthe JND image quality metric. Areference sequence is suppliedto the system under test from asource such as a video recorder

or other picture generatingequipment (providing a definedvideo quality). Objective mea-surements of picture quality

including temporal aspects ofthe human visual system should

be possible with about twoseconds of video sequence.However, subjective assessment

by an expert viewer may also bedesired so the test sequencesource should provide five ormore seconds of continuousvideo which may be repeated or

palindromed for longer viewing.At the system output, thedegraded image is captured inthe picture quality measurementinstrument which also has acopy of the reference sequence.Reference and degraded picturefiltering, differencing and datapooling is accomplished withextensive compute power andthe results made available by anappropriate human interface orcomputer data connection.

Input to the system under test isa number of short referencesequences used in a directmeasurement technique, that is,

actual pictures are used ratherthan test signals. Multiple teststimulus is also the approachfor analog or full bandwidthdigital systems which use anumber of test signals in eitherdirect or indirect measurements.For picture quality measure-ments, the different referencesequences will represent various

applications for the system andtypes of program material. Someexamples are: sports followingthe action with backgroundmoving, sports stationarycamera with the action moving,scenes with high detail, panningand zooming on high detailscenes, rotary motion with colorsnot easily handled by somecompression systems, subtleskin tones and lighting, andscenes with variable amounts ofnoise content. One requirement

is the test material be such thatthe system being measured isworking at or near the limits ofits capabilities. This has always

15

5+ SecondsVisual

Sequence

Copy of Reference

Sequence

2 Seconds ofReferenceSequence Test

Results

ComputeEngine

Degraded Test Frames

Transport DecoderEncoder

TestSequence

Source

Figure 12. Objective measurement system.


18/20

been done with traditionalanalog measurements (an exam-ple would be use of the 2-Tpulse) and is even more impor-tant to stress the non-linearcharacteristics of video com-pression systems. Althoughscenes that break either thecompression system or measure-ment method will be of some

interest to find the outer limitsof the system, they are notappropriate for repeatable andconsistent measurements.

Studies which compare subjec-tive and objective picture qualitymeasurements generally concludethere is a moderately wide vari-ation in subjective results. Thisconclusion is often emphasized

by one or more scenes whosesubjective quality does notprovide good correlation withobjective measurements.Certainly it would be desirableto develop an objective methodwith no algorithm-breakingscenes, however standardizationof well behaved and truly repre-sentative scenes should providevery useful results. Consideringthat some program material doesnot correlate with signal qualitytest results in todays analogsystems (striped shirts near thesubcarrier frequency) and thatobjective tests for compressedvideo systems are predicted to

be only 90% to 95% accurate, itwould seem appropriate for theindustry to agree on a variety of

standardized motion sequencesfor objective measurement ofpicture quality. This will allowdevelopment of very useful, ifnot perfect, picture quality mea-surement equipment. An ANSIstandard defines a number ofscenes for testing of video con-ferencing systems15 and thereare a number of standards orga-

nizations working on definitionof a wider variety of test scenes.It will be very important to havea set of standardized test scenesso measurement data willcorrelate between differentmanufacturer's test equipmentand all systems designed forsimilar applications.

In order to make objective picturequality measurements, it isnecessary to insure the two videosequences are presented to theimage quality metric calculationin much the same manner asrequired for subjective tests.That is, gain and dc level of

both the luminance and chromi-nance must be closely matched.In addition, there must be tem-poral alignment and very accu-rate spatial alignment. Theselatter two requirements are dueto the need to do a type of a dif-ferencing process between videoframes as done with PSNR andthe JND image quality metric

model. Spatial alignment to anaccuracy of one-twentieth of apixel is required. Format con-

version may be required as partof the matching process. Manycompression systems have ana-log composite NTSC or PALinputs and/or outputs. Sincecomposite encoding and decodingproduces artifacts in the picturewhich are independent of thecompression system (althoughthey may well affect operation

of the compression coder) thereare two further requirements forthe picture quality measurementinstrument: an excellent qualitycomposite decoder and a refer-ence sequence that includes thecomposite artifacts. Experimentsconducted at Tektronix haveindicated that picture qualitytesting where composite encodeand decode processes areincluded will tend to mask mea-surements of compressionsystems with small amounts of

degradation, such as, MPEG-2main profile @ main level with

bit rates in the 12 Mb/s to 15Mb/s range. This appears to be areasonable result since those bitrates represent the highest qualityof entertainment video, eitherperfect NTSC/PAL or very goodcomponent video. Systems thatdont incorporate compositesignals and provide a Rec. 601input/output can be evaluatedfor very small picture degrada-tions (suitable for studio program

production contribution quality)

16

15ANSI T1A1.801.01-1995, "Digital Transport of Video, Teleconferencing/Video Telephony Signals, Video Test Scenes for Subjective and Objective Performance Assessment".


19/20

based on the JND imagequality metric.

Use of specific reference scenesmeans that testing will be out-of-service. This paradigm forvideo testing will not be popularwith those who have, for manyyears, used vertical interval testsignals (VITS). Although in-service testing with the actualprogram material would belogistically possible in someapplications (monitoring adirect broadcast satellite systemat the up-link location) it mightnot provide meaningful resultsfor a majority of the programmaterial which does not stressthe system. Beyond that is anoperational parameter that maynot be satisfactory with generalprogram material. Time to makethe measurement is an important

feature in test equipment. If thepicture matching; gain, spatialalignment, etc., is to be doneon program material, a largeamount of compute time will berequired to make correlationcalculations. This is in additionto the time required to just makethe measurement after the twovideo scenes are correctly

matched. Therefore, it is proposedthat some known alignmentsignals, or calibration stripes, beadded to the video sequencesfor rapid picture matching asshown in Figure 13. It is expectedthat future advancements incompute power and measure-ment algorithm optimizationwill allow in-service testing forapplications where the reference(input) and degraded (output)video is available at the mea-surement instrument. This is

important for statistically multi-plexed encoding systems where

bit rates are shared betweenprograms with the potential thatany part of program could bestressful to the encoding processdue to the bit rate allowed.

Picture Quality MeasurementInstruments

The need for objective measure-ment of picture quality (degra-

dation with respect to a reference)is well established and immediate.Formal and informal subjectivepicture quality assessment has

been used to develop, test,install and operate todays com-pressed television systems. Inthis guide, we have emphasizedthe continuing need for tradi-tional test methods and

described the new methodsbeing proposed for objectivemeasurement of picture quality.Tektronix and Sarnoff Labs arecooperating on the developmentof a picture quality measure-ment product based on the JNDimage quality metric and thesignal processing required foroperation within the completemodern television system.

This is an exciting new mea-surement paradigm for the tele-

vision and telecommunicationsindustries. Please contactTektronix to express your interestin this technology so you can beinformed as more technical andproduct information becomesavailable. Further theoreticalinformation and experimentdata will be disseminated byrevisions of this guide, paperspresented at conferences,informational seminars andpublication of articles in journalsand magazines.

17

Figure 13. Calibration stripes (top of picture).


20/20

Copyright 1997, Tektronix, Inc. All rights reserved. Tektronix products are covered by U.S. and foreign patents, issued and pending.Information in this publication supersedes that in all previously published material. Specification and price change privileges reserved.

For further information, contact Tektronix:

World Wide Web: http://www.tek.com;ASEAN Countries (65) 356-3900; Australia & New Zealand 61 (2) 888-7066; Austria, Eastern Europe, & Middle East 43 (1) 7 0177-261; Belgium 32 (2) 725-96-10;Brazil and South America 55 (11) 3741 8360; Canada 1 (800) 661-5625; Denmark 445 (44) 850700; Finland 358 (9) 4783 400; France & North Africa 33 (1) 69 86 81 08; Germany 49 (221) 94 77-400;Hong Kong (852) 2585-6688; India 91 (80) 2275577; Italy 39 (2) 250861; Japan (Sony/Tektronix Corporation) 81 (3) 3448-4611; Mexico, Central America, & Caribbean 52 (5) 666-6333;The Netherlands 31 2356 95555; Norway 47 (22) 070700; Peoples Republic of China (86) 10-62351230; Republic of Korea 82 (2) 528-5299; Spain & Portugal 34 (1) 372 6000; Sweden 46 (8) 629 6500;Switzerland 41 (41) 7119192; Taiwan 886 (2) 765-6362; United Kingdom & Eire 44 (1628) 403300; USA 1 (800) 426-2200

From other areas, contact: Tektronix, Inc. Export Sales, P.O. Box 500, M/S 50-255, Beaverton, Oregon 97077-0001, USA (503) 627-1916

a guide to picture

Documents