a short introduction to multimedia forensics the science discovering the history of multimedia...

A short introduction to Multimedia

Forensics: the science

discovering the history of

multimedia contents

Sebastiano BattiatoDipartimento di Matematica e Informatica,

Università di Catania

Image Processing LAB – http://iplab.dmi.unict.it

[email protected]

http://iplab.dmi.unict.it/

Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015

Forensic science (often shortened to forensics) is the practical

application of science to matters of the law. Use of scientific

methods for gaining probative facts (from physical/analog or

digital evidences )

History of Digital Forensics• Earliest notion of digital forensics came when the Federal

Rules (US) of evidence first started to discuss digitalevidence in the 1970s

• Real digital forensics investigations started in the mid-to late1980s when federal agents had to start figuring out ways tosearch computers for digital evidence

• This “home-grown”, bottom-up approach continued until thelate 1990s when security researchers at universities andlabs started to figure out that this problem was big enough towarrant investigation.

• First research groups started around 2000 or 2001.


The first Digital Forensics Research

Workshop (DFRWS) was held in Utica, NY in

August 2001.



Multimedia

Forensics

- Source identification

- Integrity verification/tampering detection

Techniques from multimedia forensics merely provide a way to

test for the authenticity and source of digital sensor data. In this

sense is not about analyzing the semantics of digital or

digitized media objects.

Multimedia Forensics (in academic

literature)

• The terms appeared in early 2000

• K. J. Ray Liu, on “Multimedia Forensics: Where

Sherlock Holmes Meets Signal Processing” Invited

talk @ ICME 2006

• Multimedia Forensics is not Computer Forensics

(2009, Bohem et al.)

• Recent Surveys on the field (Stamm 2013, Piva 2013)

• WIFS, TIFS, Dedicated workshop at major conference

(ACM MM, ICME, ICIAP), Special issues on Int. Journal


Multimedia Forensics (in practice)

• Source Identification

• Integrity/Authenticity

• Enhancement/Restoration

• Interpretation and Content Analysis– Plate Recognition

– Dynamic Reconstruction (car crashes, etc.)

– Antropomethric issues

– …


“Forensics Image (Video) analysis is

the application of IMAGE SCIENCE

and DOMAIN EXPERTISE to interpret

the content of an image or the image

itself in legal matters” (SWGIT –

www.fbi.gov)



Recent documents:

• 2015-09-29 SWGDE Best Practices for Photographic Comparison for AllDisciplines

• 2015-09-29 SWGDE Image Processing Guidelines

• 2015-09-29 SWGDE Proposed Techniques for Advanced Data Recoveryfrom Security Digital Video Recorders v1-1

• 2015-09-29 SWGDE Training Guidelines for Video Analysis, ImageAnalysis and Photography V1-1

https://www.swgde.org/

https://www.swgde.org/documents/Released For Public Comment/2015-09-29 SWGDE Best Practices for Photographic Comparison for All Disciplines




Fantasy/Fiction


CSI


Reality


I Need That Plate! No Way...

Fantasy

• We cannot introduce new information

(there is no data..)

• But we can enhance and extract only

something already present


• What is the minimum quality for video? “Minimum quality”

doesn’t exist.

• The success of the enhancement depends on several

factors:

Main goal (video captured with an HD camera but the

license plate we need to extract is too far away)

Technical related details: Resolution of the area of

interest, Level of compression, Presence of blur /

focus, Number of available frames, Noise / brightness

and contrast

• It’s important to understand which defects are

present in order to apply the proper tools.

Understand When It’s Possible To Get

Something


Perspective Correction

Geometrical transformation


Multimedia Forensics is based on the idea

that inherent traces (like digital fingerprints)

are left behind in a digital media during both

the creation phase and any other

successively process.


• Example:• Forensic analysis of a smartphone: which pictures have been generated

on the device and which ones have been generated by other devices

and sent by messaging application or saved from the internet

• We can identify:• Type of device

• Maker and model

• Specific exemplar

Camera BallisticsWhich Device Has Created This Picture?


Source Identification Noise Based

Sensor output carries not only pure signal

but also various noise components. Sensor

noise model could be used as a

representative feature for cameras.


Sensor Identification Using

Pattern Noise


Sensor Identification Using

Pattern NoiseThis method provide good results, and is

quite reliable also using:

–images with different level of JPEG

compression (low, medium and high)

–images processed using point-wise operator

such as brightness/contrast adjustment or

gamma correction.

–images acquired by two cameras of the same

brand and model.


Integrity: What is a Forgery?• “Forgery” is a

subjective word.

• An image canbecome a forgery

based upon the

context in which

it is used.

• An image altered for fun or someone who has taken an badphoto, but has been altered to improve its appearancecannot be considered a forgery even though it has beenaltered from its original capture.


What is a Forgery?

• The other side of forgery are those who perpetuate aforgery for gain and prestige

• They create an image in which to dupe the recipient intobelieving the image is real and from this be able to gainpayment and fame

• Three type of forgery can be identified:•An image that is created using graphical software•An image where the content has been altered•An image where the context has been altered


The content has been alteredDuping the recipient into believing that the objects in animage are something else from what they really are!

November 1997: After 58 tourists were killed in a terrorist attack at thetemple of Hatshepsut in Luxor Egypt, the Swiss tabloid Blick digitally altereda puddle of water to appear as blood flowing from the temple.


The context has been altered

• Objects are be removed or added,for example, a person can beadded or removed

• The easiest way is to cut anobject from one image and insertit into another image – imageediting software makes this asimple task

• An example is this altered imagewhich could be used to influenceevents in foreign countries whichare not aware of manipulation.


Altering Images

The concepts have movedinto the digital world byvirtue of digital camerasand the availability ofdigital image editingsoftware

The ease of use of digital image editing software, which doesnot require any special skills, makes image manipulation easyto achieve.

circa 1860: This nearly iconic portrait of U.S. President Abraham Lincoln is a composite of Lincoln's head and the Southern politician John Calhoun's body.

27


Off-side (February 2011)


Barcelona – Athletico Bilbao

Sydney conman

who photoshopped

pictures of himself

with famous people

gets 12 years’jailhttp://www.smh.com.au

31


http://www.smh.com.au/

More (and more) examples

Photo Tampering through History

http://www.fourandsix.com/photo-tampering-history/

Photoshopdisaster

http://www.photoshopdisasters.com/


http://www.fourandsix.com/photo-tampering-history/

http://www.photoshopdisasters.com/

http://petapixel.com/2013/05/08/how-photographers-photoshopped-their-pictures-back-in-1946/Multimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015

http://petapixel.com/2013/05/08/how-photographers-photoshopped-their-pictures-back-in-1946/

Image Editing (2)

Malicious image editing alters the image semantic

content, mainly:

Adding information

Removing information


Piva 2013

Image Editing (2)• Splicing (two images)

– Also called cut and paste, compositing

– Used to add information

• Cloning (single image)

– Also called copy and paste, copy move, region duplication

– Used to add or remove information

– Can be exact, or the clone can be resized, rotated…)

• Inpainting (kind of intelligent clone)

– Seam carving, content aware resize, content aware fill, content dependent crop

– Used to remove information

• Retouch (local editing)

– Dodge and burn, healing tool…

• Image enhancement/filtering

– Histogram equalization, contrast enhancement, median filtering, denoise, smooth…

• Image editing (geometric transformation)

– Resize, crop, zoom, shear


Analog Vs. Digital• Analogue evidence (audio, photo, video)

– Has always an original (negative or positive)

– Copies are never an exact replica of the original

– Copies of copies will be further deteriorated

• Digital evidence

– Only a finite sequence of number

– Exact copies can be made

– No loss of information between generations

– Every copy of a digital data can be considered the original,

unless is tagged in order to be distinguished by copies

Best evidence clause (Federal Rules of Evidence 1003): if data is stored

in a computer or similar device, any printout readable by sight, shown to

reflect the data accurately, is an “original”.36


Original File: Special Cases

• Recapture: create a fake and then take a

picture with the camera we want to

pretend the picture was taken with

• Staging: the image file is authentic, but

the content has been staged

In these cases an authentic file does not

imply an authentic content.37


How To Authenticate An Image?

• Visual Inspection

• File AnalysisFile Format and Structures

Metadata (EXIF)

Compression Parameters (Quantization

Tables)

• Global AnalysisPixel and compressed data statistics

• Local AnalysisFinding inconsistencies of pixel statistics

across the image


Image Forensics Methods

Active Methods: Hiding a mark or amessage in a picture when it is created:

WATERMARKINGDrawbacks:• limitate to specially equipped

(=expensive) digital cameras;• not so robust.


Image Forensics Methods

Passive Methods: Using the alterations ofthe underlying statistics produced by digitalforgeries on an image:

PHYSICS BASEDCAMERA BASED

PIXEL BASEDGEOMETRIC BASED

FORMAT BASEDMultimedia Truthfulness Verification in Legal Environment and Social Media - WIFS 2015

INTERPOLATION

LENS CFA SENSOR

POST PROCESSINGDIGITAL IMAGESTORAGE

Processing and Storage

ORIGINALIMAGE

Acquisition


Types Of Analysis: Scene Level• Use features of the scene, not of the image pixels and could work well also

on scanned photos

• Difficult to spoof

– Most forgeries contains slight errors not visible by human eye but detectable by

proper analysis

• Difficult to automatize the analysis

– Requires a lot of experience

• Better results on splicing, less successful on forgeries which are hiding

something

• Physics based:

– Lighting

• Geometry and perspective inconsistencies

– Principal point analysis

– Shadows

– Photogrammetry


Types Of Analysis: Signal Level

Based on statistical features of pixel values; need good quality image

• Clone detection

– Cloned image blocks

– Similar couples of key points

• Resampling detection

– For resize, rotate, but also when splicing or cloning

• Enhancement Detection

– Specific for algorithms (median, histogram equalization, color adjustment)

• Seam carving detection

• General intrinsic footprints

• Inconsistencies from acquisition and coding fingerprints

– CFA, PRNU, DCT, ELA…


Format-BasedJPEG compression engine

(for both luminance and chrominance channels):

the input image ispartitioned into 8x8non-overlapping blocks


A JPEG image (luminance component only)

Format-Based


Format-Based


Format-BasedJPEG compression engine

(for both luminance and chrominance channels):

the input imageis partitionedinto 8x8 non-overlappingblocks

a DCT transformis applied to eachblock

Format-BasedJPEG forgery engine

THE TYPICAL PIPELINEFOR A COPY-PASTE

OPERATION

+

=

original image

QF(1) = q1

resulting image

QF(3) = q3

2nd image

QF(2) = q2

duplicating

resizing

Order of quantization

Top row: single quantized with steps 2 (left) and 3 (right).

Bottom row: double quantized with steps 3 followed by 2 (left), and 2

followed by 3 (right)

F. Galvan, G. Puglisi, A. R. Bruna, S. Battiato, First Quantization Matrix Estimation from Double Compressed JPEG Images, IEEE Transactions on Information Forensics and Security, 2014.



Current Trends And

Challenges

Current Trends: Point&Shoot

and Share…

Social MF on Facebook


Preliminar studies involving:

- Different devices

- Dataset of images:

- Different Scene (outdoor artificial,

outdoor natural, indoor)

- Different Quality (resolution and

compression)

- Different upload setting

has proven that some invariance could be

guaranteed.

Some specific editing could be then traced

and used to retrieve useful info about

image before uploading.

Moltisanti, Paratore, Battiato, Saravo - Image Manipulation on Facebook for Forensics

Evidence – ICIAP 2015, LNCS 2015;


Social (Multimedia) Forensics

• Image and Video Phylogeny

ReVeal project

Video Fingerprint

• Photo DNA

(http://www.microsoftphotodna.com/ )

• VideoGenome

(http://v-nome.org/ )

• Videntifier

(http://www.eff2.net/ )


73

http://www.microsoftphotodna.com/

http://v-nome.org/

http://www.eff2.net/


PRNU on Video (online)Each video is re-encoded (compressed) after it is uploaded. This

compression is needed to reduce the bandwidth usage by reducing the file

size and thus increasing streaming speeds.

By re-encoding the video, the video is altered which makes it harder to

identify the source.

Related performances depends on the brand of the camera, the resolution

and the amount of compression.

The reliability of the PRNU pattern decreases exponentially if the

compression becomes higher.

Challenges:

- Video obtained by some post processing (e.g., transition effects, fusion of

more than one source, …)

- No reference available

- Various Encoders

Who Cares?

media

Who Cares?

geopolitics…

…and political propaganda

WorldPhotoPress 2012: faked?



When is an image fake, and when

is it merely enhanced?

WPP Report:The integrity of the

Image (Nov. 2014)


Current practices and accepted

standards relating to the

manipulation of still images in

photojournalism and documentary

photography.

Media organizations

prohibit the alteration of

images beyond

traditional darkroom

techniques.

It means – as first – that the

alteration of images – where

alteration means the digital

addition or subtraction of

elements is forbidden

Data Manipulation on Science: Images

Gilbert, N. “Science Journals Crack down on Image Manipulation.” Nature (2009):

doi:10.1038/news.2009.991.

Source: Office For Research Integrity (ORI)

Forgery on Biomedical Images

Corriere della Sera – Ottobre 2013


Plagiarized paper:Fabrication of Monodisperse Magnetic Fe3O4-SiO2 Nanocomposites with Core-Shell Structures Hua Fang,* Chun-yang Ma, Tai-li Wan, Mei Zhang, and Wei-hai Shi J. Phys. Chem C 2007, 111, 1065-1070

Original PaperOriented Assembly of Fe3O4 Nanoparticles into Monodisperse Hollow Single-Crystal Microspheres Yu et al, J. Phys. Chem. B 2006, 110, 21667-21671 (Figure 3)

M. Rossner and K. M. Yamada, “What’s in a picture? The temptation of image manipulation.,”

J. Cell Biol., vol. 166, no. 1, pp. 11–5, Jul. 2004.

Forgery on Science

Forgery on Science

“What’s in a picture? The temptation of image manipulation.,” J. Cell Biol., vol. 166, no. 1, pp.

11–5, Jul. 2004.

Body-Worn camera

Image Manipulation: Case “Mozzarella Blu”

Multimedia Truthfulness

Verification in Legal

Environment and Social


Multimedia Truthfulness

Verification in Legal

Environment and Social


Evidence on the web



http://revealproject.eu/

http://www.rewindproject.eu/

http://maven-project.eu/#_=_

https://s-five.eu/

The first public draft of the FIVE Best

Practice Manual is publically available

from Oct. 9, 2015 ("October/DIWG2015

version"): DRAFT_BPM_FIVE_20151009

http://sftp.s-five.eu/Root/BPM/DRAFT_BPM_FIVE_20151009.doc

Future of Imaging


Nikon

Sharing


Conclusions

• Multimedia Forensics is now a

consolidated field but new intriguing

challenges emerge every day.

• The various involved actors are expected:

– To disseminate «basic knowledge»

– To share «best practices» and results

– To identify new challenges

– ..


Survey

• Matthew C. Stamm, Min Wu and K. J. Ray Liu,

Information Forensics: An Overview of the First

Decade (2013), in: IEEE Access, 1(167-200)

• Alessandro Piva, An Overview on Image Forensics

(2013), in: ISRN Signal Processing, 2013 (Article ID

496701, 22 pages)

- C. Baron - Adobe Photoshop Forensics – Sleuths,

Thruts, and Fauxtography – Thomson Course

Tehcnology - 2009


On line ResourcesTutorial by Prof. Hany Farid - Digital Image Forensics:

lecture notes, exercises, and matlab code for a survey

course in digital image and video

forensics. http://www.cs.dartmouth.edu/farid/downloads/tutor

ials/digitalimageforensics.pdf

Special Issue on Multimedia in Forensics, Security and

Intelligence - IEEE Multimedia Magazine Vol. 19, Issue 1,

pp. 17-19, 2012

SOFTWARE: Amped5, Authenticate, Adroit, Four&Six, Izitru,

Ghiro, …


http://www.cs.dartmouth.edu/farid/downloads/tutorials/digitalimageforensics.pdf

Credits

• Amped (www.amped.it)

• Undergraduate course: Computer

Forensics – University of Catania

(http://www.dmi.unict.it/~battiato/CF.html)


http://www.amped.it/

Prof. Sebastiano Battiato

Dipartimento di Matematica e Informatica

University of Catania, Italy

Image Processing LAB – http://iplab.dmi.unict.it

[email protected]

http://iplab.dmi.unict.it/

mailto:[email protected]

Main Contacts

Further Info

Image Processing Lab

Università di Catania

www.dmi.unict.it/~iplab

Email

[email protected]


a short introduction to multimedia forensics the science discovering the history of multimedia...

Technology