chapter 3. advanced television research program

Chapter 3. Advanced Television Research Program


Academic and Research Staff

Professor Jae S. Lim, Professor William F. Schreiber, Giovanni Aliberti, Giampiero Sciutto

Graduate Students

Matthew M. Bace, David M. Baylon, Ibrahim A. Hajjahmad, Steven H. Isabelle, DavidPeter A. Monta, Julien Piot, Ashok C. Popat, Kambiz C. Zangi

Technical and Support Staff

Francis M. Doughty, Hassan Gharavy, Cynthia LeBlanc, Julia Sharp

Kuo,

3.1 Advanced TelevisionResearch Program

The present television system was designednearly 35 years ago. Since then, there havebeen significant technological developmentshighly relevant to the television industries.For example, advances in the very large scaleintegration (VLSI) technology and signalprocessing theories make it feasible to incor-porate frame-store memory and sophisticatedsignal processing capabilities in a televisionreceiver at a reasonable cost. To exploit thisnew techology in developing future tele-vision systems, Japan and Europe establishedlarge laboratories, funded by government orindustry-wide consortia. The lack of thistype of organization in the U.S. was consid-ered detrimental to the broadcasting andequipment manufacturing industries, and, in1983, a consortium of U.S. companies estab-lished the Advanced Television ResearchProgram (ATRP) at MIT.

Currently, the consortium members includeABC, Ampex, General Instruments, Kodak,Motorola, NBC, NBC Affiliates, PBS,Tektronix and Zenith. The major objectivesof ATRP are:

* To develop the theoretical and the empir-ical basis for the improvement of existingtelevision systems, as well as the designof future television systems.

* To educate students through television-related research and development and tomotivate them to enter careers intelevision-related industries.

* To facilitate continuing education of sci-entists and engineers already working inthe industry.

* To establish a resource center for dis-cussion and detailed study of problemsand proposals

* To transfer the technology developedfrom this program to its industrial spon-sors.

The research areas of the program includethe development of transcoding methods andthe design of (1) a channel-compatibleadvanced television (ATV) system, (2) areceiver-compatible ATV system, and (3) adigital ATV system. We have already madesignificant advances in some of theseresearch areas. We have designed a channel-compatible ATV system, scheduled for testingin 1991 by the FCC for its possible adaptionas the U.S. HDTV standard for terrestrialbroadcasting.

3.1.1 ATRP Computer Facilities

The main ATRP computer facility currently isa VAX 11/785 minicomputer with approxi-mately 2.4 GBytes of online disk space.Attached to the VAX is a DATARAM WideWord Storage system which provides 320MBytes of RAM. The high speed interface tothe Wide Word system drives a three-dimensional interpolator that was con-structed by graduate students in the lab. Thethree-dimensional interpolator can performseparable spatiotemporal interpolation. The

257


output of the interpolator feeds a custombuilt data concentrator which drives a Sony2k by 2k monitor, running at 60 frames/sec.

In addition to displaying high resolution realtime sequences, the ATRP facilities include a512 by 512 Rastertek frame buffer and anNTSC encoder. The Rastertek frame bufferfeeds static images to nearly a dozen moni-tors that are distributed throughout the lab.The NTSC encoder allows us to record theresults of sequence processing onto either3/4 inch or VHS tape. For hard copy output,the lab uses an Autokon 8400 graphicsprinter for generating high resolution blackand white images directly onto photographicpaper.

For preparing presentations, ATRP alsoemploys a Macintosh SE30 microcomputerand a Mac Ilx, feeding an Apple LaserWriter.

To support the growing computation needsof the group, three Sun-4 workstations willbe installed in the near future. They willhave 24-bit color displays, local disk storage,and DSP boards to assist with computation-intensive image processing.

A fast network (FDDI) is under considerationto link the machines and to display devicessuch as the Dataram. The workstations willalso support Ethernet and will be connectedto Internet through the building subnet.

3.1.2 Receiver-Compatible AdaptiveModulation for Television

Sponsors

National Science FoundationGrant MIP 87-14969

National Science Foundation Fellowship

Project Staff

Matthew M. Bace, Professor Jae S. Lim

There have been numerous proposals fordeveloping methods to improve the quality ofthe current NTSC television picture. Most ofthese proposals have concentrated onmethods for increasing either the spatial orthe temporal resolution of the televisionpicture. While these proposals promise sig-nificant improvements in picture quality, until

an effective scheme to combat channel noisehas been introduced, these improvementswill never be fully realized. Degradationssuch as random noise ("snow"), echo, andintersymbol interference (channel crosstalk)are still the greatest barrier to high-qualitytelevision.

This research will attempt to develop areceiver-compatible scheme to reduce theeffects of channel imperfections on thereceived television picture. In particular, themethod of adaptive modulation will beemployed to make more efficient use of thecurrently under-utilized bandwidth anddynamic range of the NTSC signal. By con-centrating more power in the higher spatialfrequencies and using digital modulation tosend additional information in the verticaland horizontal blanking periods, we canmake existing television signals more robustin the presence of high frequency disturb-ances. Furthermore, we can adjust theparameters of this scheme so that the modu-lated signal may be received intelligibly evenon a standard receiver (although an improvedreceiver will be required to realize the fullbenefits of adaptive modulation).

Before we can conclude which adaptivemodulation schemes are optimal, we mustconsider many details. Among the parame-ters which can be varied are: (1) the controlover the adaptation and compression factors,(2) the form of the input low-pass filters, (3)the interpolation scheme to be used at boththe transmitter and receiver, and (4) theencoding of the digital data. We will adjustthese parameters to optimize the performanceof the modulation scheme with repect to twofundamental performance criteria. These cri-teria are the degree to which (1) the channeldegradations are removed when the signal isreceived on an improved receiver and (2) thesignal is distorted when received on astandard receiver.

3.2 Adaptive AmplitudeModulation for TransformCoefficients

SponsorAdvanced Television Research Program

258 RLE Progress Report Number 132

Project StaffDavid M. Baylon, Professor Jae S. Lim

In this project, we have shown that adaptiveamplitude modulation/demodulation(AM/DM) is an effective noise reductiontechnique. However, this technique requirestransmitting the adaptation factors as sideinformation. It is important to minimize therequired side information in systems thathave limited transmission bandwidth. Ourresearch will focus on representing the adap-tation factors by a few parameters exploitingproperties of the signal in the transform (fre-quency) domain.

Previous investigations of adaptive amplitudemodulation have been based on time domainmethods. Specifically, in two-dimensionalsubband filtering, an image is decomposedinto a set of spatial frequency subbands thatare adaptively modulated. Similarities amongsubbands are exploited by reducing thenumber of adaptation factors to about one-sixth the number of data points. Neverthe-less, further reduction in the amount of sideinformation is desirable.

This research will take a different approach toreducing the amount of side information byadaptively modulating the transform of thesignal. Transform coefficients of typicalimages tend to decrease in energy away fromDC. By exploiting this property, we canmodel the transform coefficients and theadaptation factors with a few parameters (forexample, an exponential model). Conse-quently, we can significantly reduce theamount of side information compared withthat required by previous methods.

Research will focus on determining the bestway to model the adaptation factors with afew parameters in systems that are band-width and peak power constrained. Perform-ance criteria of the various AM/DM schemeswill include signal-to-noise ratios and overallimage quality. Among the many ways ofobtaining the coefficients (such as using


subband filtering or the lapped orthogonaltransformation (LOT)), the discrete cosinetransformation (DCT) will be used becauseof its many desirable properties, includingcoefficient uncorrelation, energy compaction,and efficient computation using the fastFourier transform (FFT). Issues that we willaddress include choosing the appropriateblock size and determining the best AM/DMmethod with an adaptive coefficient selectionscheme (such as used in image codingsystems). We will study both two-dimensional images and three-dimensionalvideo.

3.3 Transform Coding for HighDefinition Television

Sponsor

Advanced Television Research Program

Project Staff

Ibrahim A. Hajjahmad, Professor Jae S. Lim

Image coding has many useful applications.One important application is for compressingchannel bandwidth for image transmissionsystems such as HDTV, video conferencing,and facsimile. Reducing storage require-ments for tasks such as digital videorecording is another important application ofimage coding.

Image coding can be divided into a numberof classes, depending on which aspects ofthe image are being coded. One class is thetransform image coder,' in which an image istransformed from the spatial domain to a dif-ferent domain more suitable for coding.Then, the transform coefficients are quan-tized and coded. When received, the codedcoefficients are decoded and then inversetransformed to obtain the reconstructedimage.

To perform transform coding one must selectan appropriate transform. In particular, the

1 J.S. Lim, Two-Dimensional Signal and Image Processing (Englewood Cliffs, New Jersey: Prentice Hall, 1990);R.J. Clarke, Transform Coding of Images (London: Academic Press, 1985).

259


Discrete Cosine Transform (DCT) 2 is veryuseful because of two important properties.First, the energy compaction property statesthat a large amount of energy is concentratedin a small fraction of the transform coeffi-cients (typically the low frequency compo-nents). Because of this property, only asmall fraction of the transform coefficientsneed to be coded, while little is sacrificed interms of the quality and intelligibility of thecoded images. Second, the correlationreduction property reduces the high corre-lation among pixel intensities in the spatialdomain. In effect, the redundant spatialinformation is not coded.

Currently, we are investigating the use of theDCT for bandwidth compression. In addi-tion, we are studying new adaptive tech-niques for quantization and bit allocation toreduce further the bit rate without sacrificingimage quality or intelligibility.

3.4 Filter Design for MultirateFilter Banks

Sponsors

Advanced Television Research ProgramAT&T Bell Laboratories Doctoral Support ProgramNational Science Foundation

Grant MIP 87-14969

Project StaffSteven H. Isabelle, Professor Jae S. Lim

Multirate filter banks have wide applicationin the areas of subband coding of speechand images. In this application, the signal ispassed through a bank of bandpass filters.The resulting bandpass signals are decimatedand then coded. At the receiver, the signalsare interpolated and added to form thereconstructed signal. In this scheme, thecoding technique can be adjusted on a fre-quency dependent basis to match theobserver's perceptual characteristics. Clearly,the performance of this kind of anal-ysis/synthesis system depends strongly onthe properties of the filters in the analysisand synthesis filter banks. For example, there

are two desirable properties: (1) in theabsence of coding, the reconstructed signalshould be nearly identical to the original, and(2) there should be little interaction betweenthe different subband signals. One goal ofthis research is to develop improved analysisand design tools for multirate filter banks. Tothis end, we have developed a design algo-rithm with improved convergence behaviorover existing methods for the special case ofa two-channel perfect reconstruction filterbank.

The subband decomposition of an image istypically performed using separable filtersalong its horizontal and vertical axis. The useof nonseparable filters allows for directionalselectivity in the subband decomposition anda potential improvement in the subjectivequality of encoded images. A second goal ofthis research is to develop design techniquesfor general multidimensional, multirate filterbanks.

3.5 Adaptive SpatiotemporalFiltering

Sponsors

Advanced Television Research ProgramKodak Fellowship

Project Staff

David Kuo, Professor William F. Schreiber

The current NTSC television standard speci-fies a frame rate of 60 fields/sec throughoutthe transmission chain. The purpose of thisframe rate is for minimizing the visibility ofannoying flicker at the display. However, toeliminate flicker, only the display mustoperate at the high frame rate; the channeldoes not need to be constrained to operatingat 60 frames/sec. Because there is a greatdeal of correlation between neighboringframes of an image sequence, a high framerate through the channel seems bandwidthinefficient.

One way to take advantage of the correlationbetween neighboring frames is to transmit

2 N. Ahmed, T. Natarajan, and K.R. Rao, "Discrete Cosine Transform," IEEE Trans. Comput. C-23:90-93 (1974).


only a temporally subsampled version of theoriginal sequence, relying on the receiver torecover the inbetween frames. However, ourprior work suggests that the receiver musthave more information than simply the sub-sampled frames. This research focuses onusing motion vectors as part of the imagesequence representation.

There are three main areas of focus in thisresearch. First, we consider the use of adap-tive spatiotemporal prefiltering as a means ofreducing the aliasing that arises fromtemporal subsampling. Secondly, we explorethe characteristics of the motion vectors.Finally, we consider how to use multipleframes of data to improve the motion esti-mation process.

3.6 Signal Processing forAdvanced Television Systems

Sponsors

Advanced Television Research ProgramU.S. Air Force - Electronic Systems Division

Contract F1 9628-89-K-0041

Project Staff

Peter A. Monta, Professor Jae S. Lim

Digital signal processing will play a large rolein future advanced television systems.Source coding to reduce the channelcapacity necessary to transmit a televisionsignal and display processing such as spatialand temporal interpolation are the majorapplications. Present-day television stand-ards will also benefit significantly from signalprocessing designed to remove transmissionand display artifacts. This research will focuson developing algorithms and signal modelsto (1) enhance current standards (bothcompatibly and with some degree of cooper-ative processing at both transmitter andreceiver) and (2) improve proposed HDTVsystems.

Using a receiver with a high-quality displayand significant computation and memory, wecan improve the American televisionstandard, NTSC, in several ways. We canremove interlace artifacts, such as line visi-bility and flicker by converting the signal to aprogressive format prior to display. We can


greatly reduce color cross-effects with accu-rate color demodulators implemented withdigital signal processing techniques. Wehave tested and implemented an algorithmfor an advanced receiver that can recover amuch improved image by exploiting structurein the film-NTSC transcoding process if filmis the original source material.

Similar ideas apply to HDTV systems. Forexample, film will be a major source materialwell into the next century, and HDTV sourcecoders should recognize film as a specialcase, trading off the inherent reducedtemporal bandwidth for better spatial resol-ution. The MIT-CC channel-compatibleHDTV system will adapt to film in this way.

3.7 Adaptive FrequencyModulation for SatelliteTlevision Systems

Sponsor


Project Staff

Julien Piot, Professor William F. Schreiber

Frequency modulation is the first choicecoding scheme in many existing applicationssuch as satellite television transmission.

A simple model of image formation predictslarge variations in the short-time bandwidthof the modulated signal. Based on thismodel, we adjust the frequency deviation in asmall block of the picture to keep the band-width constant. We show that the resultingnoise improvement is significant when weuse a subjective measure of the transmissionerror. This measure, based on noise masking,has an average intensity related to the blockstatistics.

In some applications the modulated signal isbandlimited, resulting in envelope and phasedistortion. Both terms generate artifacts inthe recovered picture, mostly when noise ispresent in the link. We show through meas-urements that the peak short-time bandwidthis related to the severity of the distortion,hence justifying the prior approach to adap-tation. We introduced improved algorithms

261


that minimize the transmission noise whilemaintaining negligible distortion.

When the information is transmitted in theform of multiple components, we present atechnique based on a sequential transmissionsuch as subbands. This technique can alsobe used to transmit some side information,such as the adaptation function of themodulator. By adjusting the rate of trans-mission of the various components, we canminimize the subjective impairment. Forexample, we show that a vertical subbanddecomposition is very effective in reducingtransmission noise, with a larger improve-ment than preemphasis techniques. Alterna-tively, adaptive modulation can be combinedwith this technique.

Finally, we applied the principle of adaptivefrequency modulation to broadcasting anddistributing television signals by satellite. Wedemonstrated a dual-in-one system, wheretwo NTSC video signals are transmittedthrough one transponder. Another systemproposes direct broadcasting by satellite ofhigh definition television using subbandcoding and adaptive frequency modulation.A simulation of the two systems demon-strates that high quality transmission is pos-sible in noisy narrow-band channels, usinganalog modulation.

This study was completed in December1989.

3.8 Subband Coding forChannel-CompatibleTransmission of High-DefinitionTelevision

Sponsor


Project Staff

Ashok C. Popat, Professor William F. Schreiber

In recent years, subband coding has receivedconsiderable attention from the image codingcommunity as a simple and effective meansof efficiently representing image and image-sequence data. 3 We have proposed a three-dimensional (horizontal, vertical, andtemporal) subband coding technique forapplication in a 6-MHz channel-compatiblehigh-definition television (HDTV) distribu-tion system.4 Although preliminary "proof-of-principle" tests have demonstrated that thetechnique is effective, the tests have alsoshown that there is considerable room forimprovement. The technique can beimproved by adjusting various parameters inthe system; these parameters include thedegree of data compression, the number ofsubbands in each dimension, the type andlength of the subband analysis/synthesisfilters, and the means of selecting subbandpixels to be retained and transmitted. Wehave observed a high degree of interdepend-ency among many of the system parameterswhich has complicated the process of identi-fying the particular combination of parame-ters that is best suited to the presentapplication. In particular, the strong interde-pendency seems to eliminate the possibilityof finding the best choice for each parameterseparately. A major objective of the presentresearch is to search through the vast param-eter space by judicious choice of parametersfor computer simulation, and by objectiveand subjective evaluation of the simulationresults.

One critical set of system parameters is theset of coefficients used in the subband anal-ysis/synthesis filter banks. We have devel-oped a novel approach to designing suchfilters based on time-domain numericalsearch; the approach is fairly general and hasresulted in critically-sampled filter banks that

3 J.W. Woods and S.D. Oneil, "Subband Coding of Images," IEEE Trans. Acoustics, Speech, and Signal Processing.34:1278-1288 (1986); H. Gharavy and A. Tabatabai, "Subband Coding of Monochrome and Color Images," IEEETrans. Circuits Syst. 35:207-214 (1988).

4 W.F. Schreiber, et al., Channel-Compatible 6-MHzHDTV Distribution Systems, CIPG Technical ReportATRP-T-79, MIT, January 1988.


are extremely well-suited to image codingapplications.5

A seemingly basic principle of imagesubband/transform coding has emerged fromthe present study. In particular, the bestchoice for the length of the anal-ysis/synthesis filters depends only weakly onthe number of subbands, depending morestrongly on the spatial extent over which theimage can be well-modeled as stationary.Thus, the nonstationarity of images inevitablyleads to an uncertainty-principle basedtradeoff in the selection of the number ofsubbands and lengths of filters.

We also found that it is extremely importantthat the allocation of channel capacity is spa-tially varying. It is essential to be able toincrease the number and/or fidelity ofsamples used in representing action regionsof the image at the expense of more poorlyrepresenting inactive regions. Currently, weare devising a fixed-rate, practicable meansof exploiting this principle.

3.9 Channel Equalization andInterference Reduction UsingAdaptive Amplitude Modulationand Scrambling

SponsorAdvanced Television Research Program

Project StaffAdam S. Tom, Professor William F. Schreiber

Terrestrial broadcast channels and cablechannels are imperfect. Random noise,multipath (ghosts), adjacent and co-channelinterference, and an imperfect frequencyresponse degrade these transmission chan-nels so that the quality of the signal at thereceiver is significantly below that at thetransmitter. To appreciate the increasedresolution of high definition images, degra-dation due to channel defects needs to bereduced. Conventional methods of channel


equalization, which use adaptive filters, arelimited by convergence time, length of filters,and computational complexity. We areresearching a new method of channel equal-ization and interference reduction basedupon the ideas of adaptive amplitude modu-lation and pseudo-random scanning (scram-bling). This new method is not bound bythe above limitations; however, it is limitedby the energy of the channel degradationsproduced.

Adaptive modulation is a noise reductiontechnique that is applied only to high fre-quency components of signals. Prior totransmission, a set of adaptation factors aremultiplied with the input signal to raise theamplitude of the signal according to thestrength of the signal. At the receiver, thesignal is divided by these same adaptationfactors. In this manner, the random noiseadded in the channel is reduced by a factorequal to the adaptation factor. The noise isreduced more in the blank areas relative tothe busy areas.

Scrambling is a technique for reducing theeffects of multipath, adjacent and co-channelinterference and also imperfect frequencyresponse. Prior to transmission, we pseudo-randomly scan the input signal, scramblingthe signal so that it appears as random noise.Then we transmit this scrambled signalthrough the channel and perform the reverseof the scrambling at the receiver. Conse-quently, any degradations in the channel arethemselves scrambled at the receiver and,thus, have the appearance of pseudo-randomnoise in the decoded signal, while thedesired signal remains sharp and in fullbandwidth.

Since the resultant signal in the receiver nowhas a noisy appearance, we apply the noisereduction technique of adaptive modulationto the input signal. In our scheme, we applyadaptive modulation to the input signalbefore scrambling. The coded signal is trans-mitted through the imperfect channel anddecoded at the receiver. The decoding con-

5 A.C. Popat, "A Note of QMF Design," unpublished memo, Advanced Television Research Program, MIT,December 1988; A.C. Popat, "Time-Domain Numerical Design of Critically Sampled Filter Banks," presentationviewgraphs, Advanced Television Research Program, MIT, October 1988.

263


sists of doing the reverse of the scramblingand then the reverse of the adaptive modu-lation. Scrambling causes any channel degr-adations to have a noiselike appearance, andadaptive modulation reduces the appearanceof this pseudo-random noise. In thismanner, the degradations to a transmittedsignal are reduced and the channel is equal-ized.

3.10 A Novel QMF DesignAlgorithm

Sponsor


Project Staff

Kambiz C. Zangi, Professor William F. Schreiber

In this research, we presented a new algo-rithm for designing two-band quadraturemirror filter (QMF) banks. We show that thisalgorithm is computationally more efficientby a factor of eight than existing algorithms.Moreover, because this algorithm is not sen-sitive to the choice of the initial guess, byusing it we can also design two-band QMFbanks in which one of the filters ispredescribed.

This work was completed in December 1989.

Professor William F. Schreiber shown with his graduate students. From the left: David Kuo, Adam S. Tom,Professor Schreiber, and Peter A. Monta.


chapter 3. advanced television research program

Documents