soima11-digital-imagingbesser.tsoa.nyu.edu/howard/talks/11soima-digital-imaging.pdf · • mpeg 2...

7/13/11

DigitalImaging—Compression&Sampling

Averyshortintroduc?on

HowardBesserNYUMovingImageArchiving&Preserva?onhHp://besser.tsoa.nyu.edu/howard/Talks/hHp://www.nyu.edu/?sch/preserva?on/

1Besser‐SOIMA2011‐Compression&Sampling

WhatisDigitalEncoding?‐

•  WhyDigitalisdifferentthanAnalog– DigitalAudioasexample

•  CompressionforAudioandS?llImages– FramesandFrameRate,BitRates,ChromaSubsampling

•  Theprocessofdigitalphotography•  DigitalCompressioncoding

•  Wrappers

•  (notdiscussingcopyphotography)2Besser‐SOIMA2011‐Compression&Sampling

Besser‐ICCROMRIO,6/8/073

SamplingSoundWaveoverTime

Frequency : every for seconds

DigitalSamplePointsonAnalogWave

DigitalAudioSamplingconvertfromcon?nuoustodiscrete

Besser‐SOIMA2011‐Compression&Sampling 5 Besser‐ICCROMRIO,6/8/076

Digi?za?on,samplingevery4seconds,1bit

Sample Time

Voltage Signal (in base 2)

0 4.3 1 4 6 1 8 6.5 1 12 1.5 0 16 4.2 1 20 4 1

Resolution

7/13/11

AudioDigi?za?onsebngsdependonseveralfactors

•  Thequalityontherecording,andthequalityyouneedforyourusers– Whenyouneedtocapturesoundsthatarelesscomplex(shorterandlesscomplexsounds),youcanloweryoursignalresolu?onand/orsamplingrate(thehumanvoiceonaninterviewrequireslesssignalresolu?on/fidelitythanmostmusic

•  Digi?za?onsebngsforspeecharerela?velylowunlessyouintendtocapturebackgroundsoundsaswell(backgroundmusicwhispers,gunshots,…)

Resolu'on

•  Humanspeechcanbeeffec?velyreproducedatarateof5.5kHz(kilo‐Hertz)

•  5500samples/sec

•  Mostnaturalworldsoundsandmediumfidelitymusiccanbereproducedat11kHzwithacceptablelossesoffidelity

•  11000samples/sec.

•  highfidelitymusicatCDaudioqualitythesamplingratemustbeabsoluteminimumof44.1kHz,givingaplaybackrateof22kHz,whichisjustabovethelimitofhumanhearing,butmostprofessionalandacademicorganiza?onsrecommendaboutdoublethat(96kHz).Butfor44.1kHZ:

•  44100samples/sec.•  22000samples/secisjustabovethelimitofhumanhearing

Quan'za'on

•  Thenumberofdis?nctsoundlevelsthatcanberepresentedisdeterminedbythenumberofbytesusedtostorethequan?za?onvalue.

•  themostcommonquan?za?onstrategyuses16bits(2bytes).Ex:11111111

•  216=65536discretelevels

Audio:Pulse‐CodeModula?on(PCM)

•  Samplingrate(numberofsamplespersecond)– 48kHz,96kHz

•  Bit‐depth(rangeofpossiblevaluesforeachsample)– 16bit,24bit

Besser‐SOIMA2011‐Compression&Sampling 10

Standards/BestPrac?cesforSoundDigi?za?on

•  Aremanagedbyprofessionalcommuni?eswithcommoninterests–  Interna?onalOralHistoryAssocia?onhHp://www.iohanet.org/resources/technical.html

–  Interna?onalAssocia?onofSoundandAudio‐VisualArchives(IASAA)hHp://www.iasa‐web.org/audio‐preserva?on‐tc04

–  SoundDirec,ons:BestPrac,cesforAudioPreserva,onh7p://www.dlib.indiana.edu/projects/sounddirec,ons/papersPresent/index.shtml

–  UniversityLibraries,…

S?llImageEncoding

7/13/11

FilmvsCCDArray

CCDforcapture

•  fromhHp://micro.magnet.fsu.edu/primer/digitalimaging/concepts/fullframe.html

Pixels

Resolu?on

•  dpi,ppi,spi(scannervs.camera)

Example:50‐800ppi

7/13/11

Bit‐depth

•  8‐bit,24‐bit,36‐bit

Bit‐depthexample

Full‐colorspectrumvs8‐bits

RGBColor

Compressionforcapturingcon?nuous‐tonescenes

•  JPEG/JFIF– Predic?vecoding– DCT‐

Compressionforcapturingcon?nuous‐tonescenes

•  JPEG/JFIF– Predic?vecoding– DCT‐

7/13/11

ChromaSubSampling/DCTencoding

•  Mostly,eyeismoresensi?vetoluminencethantochroma

•  Divideimageintomatrix(8x8cellblocks;moreblocksforH.264[butalsoincludesub‐blocks])

•  Separateluminence(Y)fromChroma(Cb,Cr)

•  Encodeluminenceforeachindividualcell,butencodechromaforgroupsofcells(chromasubsampling)

Encodemoredataforluminencethanchroma

•  4:4:4isfullysampled

•  4:2:2useshalfthechromasamplesoneachchromachannel

•  4:2:0uses¼thechromasamplesoneachchromachannel

‐ImagefromTonyGladvinGeorge

DCTCodingar?facts

Ar?factsbecomelessvisibleaswezoomout

FileFormats

•  TIFF•  JFIF(JPEG)•  GIF•  J2K•  Raw•  DNG(Adobe)

7/13/11

Differentfileformats

OtherAr?facts:Aliasing

•  Aliasingduetohighfrequencyclearedupwithlow‐passfilteringbeforedigi?zing

‐hHp://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/sab/report.htmlBesser‐SOIMA2011‐Compression&Sampling 32

CameraMetadata

•  XMP(Adobe’seXtensibleMetadataPlaworm)•  IPTC(Interna?onalPressTelecommunica?onsCouncil)–  hHp://www.iptc.org/site/Photo_Metadata/Embedded_Metadata_Manifesto_%282011%29

•  EXIF(ExchangableImageFileFormat)‐

EXIFmetadataexample

DigitalVideoCompression:Codecs

•  theschemeusedforcompressingandencodingthematerial.Usuallyrequiresdecodingsoywareorhardwaretoplayit.–  Hardwaredecodingindigitalvideotapeplayers–  Usuallysoywaredecodinginfile‐baseddigitalvideo

•  withwebbrowsers:mightbebuiltintobrowser(jpg)orrequirebrowserplug‐in(flv)

•  Otherwise,requiresplayersoywareorhardwarecard(J2K)•  Encodingisusuallyasymetrical

–  Takeslotsof?me&resourcestoencode

– …butdecodingcanbedonein(near)real?meonstandardwkst

MovingImageDigitalFiles

•  Eachframe(25‐30fps)isessen?allystoredsimilarlytoas?lldigitalimageframe(intermsofrepresenta?onandcompression)

•  Compressedformatssaveonstoragespacebynotsavingeverysingleframe– Maystoreonlythesmalldifferencesfromthepreviousframe

– Maystoreperiodicfullframesandpredict/interpolatebtwnthestoredframes

7/13/11

DigitalVideoCompression

•  Spa?alimagecompression(intraframe)•  Temporalmo?oncompression(interframe)

•  Andmostcompressionschemesgiveyouarangeofchoicesinhowheavilytocompress

Differentcompressionlevels(CarlFleischhauer)

Temporalmo?oncompression(interframe)‐basics

Interframecompression‐‐moredetails

•  I‐Frame(keyframe)—encodedass?llimage•  P‐Frame/Delta‐Frame—differencesfrompreviousframe

•  B‐Frame—a“virtual”framethataveragesdifferencebtwnframesbeforeandayer

‐imagefromPeHeriAimonenBesser‐SOIMA2011‐Compression&Sampling 40

Intra‐FrameCompression InterframeCompression

•  I‐Frame(orkeyframe)isonlyframethatdoesn’tdependuponotherframestodisplayit,butittakesupthelargestamountofstorage/bandwidth– NeedI‐Frametorecoverfromproblems

•  IntervalsbtwnI‐Frames(4,6,8,…,150)canbespecifiedinH.264orFLVformats

7/13/11

EncodingAr?facts&RelatedIssues

•  Ifevenonebitgoesbad,onecanlosesignalun?lnextkey‐frame(because,withinter‐framecompression,alltheframesbeforethenextkey‐frameonlycontainpar?alinforma?on)

•  Browsingwillonlyfindkey‐frames

•  Canseeablockingeffectofchromasubsamplingaroundthe?meoflumainterference‐

CodingAr?facts

"JensHenneman”,PerformanceArtpiecebyFredArmisen)careofPeterOleksik.720x980HDvideo,compressedusingMPEG2codec(23.98fps@2mbit/s),withonepassCBR

CodingAr?factsfollowcolorlightflash

Tamiment‐35mmtelecinedtoDigibeta,thentranscodeduncompressedintocomputer.Then,filewascompressedusingMPEG2codec(3.7mbit/s)

AspectRa?oissuesTerryGilliam’sBrazil,digi?zed4:3VHSusingaDV50codecs?ll

displayedwithbadaspectra?o

7/13/11

VideoDigi?za?on&Transfer Framesize&rate

•  Framesize‐‐pixelarrayplusra?o(4:3,16:9)•  Framerate(29.97fps,24fps,…)

•  Somedigitalcompression&storageformatsrequireaspecificframesize

BitRate

•  Datatransferrate‐‐Amountofdatapersecond(comingacrossanetwork,orfromdisk/tapetoscreen)

•  UsuallymeasuredinMb/sec(8Mbps,30Mbps,50Mbps,200Mbps)– oyennotedas30Mbps(uncompressed),or8Mbps(compressed)

• AVI Microsoft format • MOV Quicktime format • MPEG compressed standardized format • RM Real Media compressed format • ASF Microsoft streaming format • DV25 25 Mbits/sec, fixed compression, 4:1:1 color

encoding, 3.6 MB per sec, 60 min. mini-DV cassette has 13 GB of storage • DV50 50 Mbits/sec, 4:2:2 or 3:3:1 color encoding • DV100 100 Mbits/sec, to be used for HDTV

production

From Steve Puglia from Adobe’s “A Digital Video Primer” and http://www.cit.cornell.edu/atc/materials/dig/videoformats.shtml

FileFormats

JimWheeler’slistofDigitalVideoFormats

•  MPEG2‐‐currentlymostcommonlyusedfordistribu?on,butdifficulttouseforproduc?on

•  MPEG4‐‐usedwithQuickTimeandWindowsMediafiles

•  MPEG7&21‐‐metadatastandards,notusedforcompression

7/13/11

Encodingformats

•  MPEG‐1,op?mizedforCD‐ROMat1.5Mb/sec,1992,usedforVCDs

•  MPEG‐2,designedfor10Mb/sec,1994,usedforDVDsandmanybroadcasttvformats

•  MPEG‐4,madeforspeedsasslowas64Kb/sec,canseparateouts?lls/audio/video,goodforanima?onandstreaming

•  H.264(MPEG‐4part10),usedforBlu‐Rayandsomebroadcasttv

•  J2k(mo?onJPEG2000)usedinDigitalCinema(DCDM)•  WMV

•  .mov•  Flash(SWF),Real,…

Encodingformatgeneralproper?es

•  FrameIntegrity‐‐DPX2andJ2kencodeindividualframesasseparatefilesordis?ncten??es

•  Mostversa?le(varietyofimplementa?onchoices)‐‐Quick?meandMPEG‐4

Transcoding

•  Canalwaysbedonewithenoughhorsepower•  Butsomequalitycanbelost,dependinguponcompression,movingbtwncolorspaces(RGBandY’CbCr),etc.

Streaming

•  Latencyisprimaryissue(whichsome?mesnecessitatescaching)

•  FormatsincludeFlash(SWF7),Real,MPEG‐4

Wrappers

•  Forwrappingboththe“essence”andthemetadataaboutitintothesamecontainer

•  Wrapperscaneitherhaveeverythingembeddedwithinasinglefile,ortheycanhavepointerstofilesoutsidethewrapper

•  Wrappersinclude:–  MXF(broadcas?ngworld,canwrapanything)

–  AAF(videoedi?ngworld,canwrapanything)–  METS(libraryworld,canwrapanything)–  Quick?me(canwrap.mov,DV,MPEG‐2/4,J2K,Flash)

–  AVI(canwrapDV,J2K,DivX,mp3)

–  ASF(proprietaryMicrosoy)wrapperforWMVBesser‐SOIMA2011‐Compression&Sampling 59 Besser‐ICCROMRIO,6/8/0760

DigitalCinema

•  24fps•  2K(2,000X1,500)perframefortheatricaldistribu?on

•  4K(4,000X3,000)perframeformasterfile(36M/frameuncompressed)

7/13/11

Howmuchstoragedoyouneed?

•  1hourvideo•  640X480•  24bitscolor•  25fps•  =83Gbytes

Administra?ngDigitalVideoover?me

•  Preserva?onofdigitalvideofilesshouldfollowstandardrepositoryprac?cesforpreservingalltypesofdigitalcontent–  OAIScompliantrepositoryarchitecture

– Metadatawrappers(includingpreserva?onmetadatasuchasPREMISthatdocumentseachchangeorreformabngorreinstalla?onortransforma?on)

–  Periodicmonitoringofcheck‐sums–  RegularRefreshing– Migra?onand/orEmula?onstrategies

ImportantGuide(1/2)

ImportantGuide(2/2)

Re/Sources

•  Besser,Howard.Talks,hHp://besser.tsoa.nyu.edu/howard/Talks/

•  Besser,Howard.Introduc?ontoImaging,hHp://www.geHy.edu/research/publica?ons/electronic_publica?ons/introimages/index.html

•  AICGuidetoDigitalPhotographyandConserva?onDocumenta?on,2ndEdi?on,hHp://www.conserva?on‐us.org/index.cfm?fuseac?on=Store.viewProduct&product_ID=19276&prod_cat_ID=13

•  Fleischhauer,CarlandCarolineArms,SustainabilityofDigitalFormats,PlanningforLibraryofCongressCollec?ons:ContentCategories‐MovingImages,hHp://www.digitalpreserva?on.gov/formats/content/s?ll.shtml

•  IMAP/EAIPreserva?onGuidehHp://www.eai.org/resourceguide/preserva?on.html

•  MIAPWebsitehHp://www.nyu.edu/?sch/preserva?on/

soima11-digital-imagingbesser.tsoa.nyu.edu/howard/talks/11soima-digital-imaging.pdf · • mpeg 2...

Documents

iprs distribuon rules and method s

lexicon lexicon lexicon lexicon 3rd semi...deﬁnion of...

imaging in rare neuroimmune...

p2p systems and distributed hash tables€¦ · distributed...

distribuon/+ marke(ng+channel+for+...

iprs distribuon rules / scheme and methodspage 1 of 22 iprs...

within the resolution cell_super-resolution in tomographic...

cassini imaging science: instrument characteristics...

integrated models for transmission & distribuon plus

polypropylene suture button promo - bioseal inc.bioseal inc....

2014 validity of measuring distal vastus medialis muscle...

detection of common defects on oranges using hyperspectral...

imaging a beam with synchrotron light - uspas | u.s...

linux standard base imaging...

cuda accelerated real time signal processing in high...

oberoi international - microsoft€¦ · timely delivery...

radiologist order entry: making imaging protocol ... order...

general abdominal imaging peritoneal space, pancreas, &...

newton corporate center - loopnet › d2 ›...

brain imaging.pdf