Computer-Aided

Technologies for

Food Risk

Assessment

Candidate: Francesco Rossi

Tutor: Prof. Alfredo Benso

XXX Cycle PhD in Computer and Control Eng.

at Politecnico di Torino

Motivation of PhD in Computer Science

Master’s Degree

Biomedical Engineering

Virtual

Colonoscopy

Prostate Cancer

Research

#machinelearning #computervision #computeraided

Computer-Aided Technologies for Food Risk Assessment

Computer-Aided Technologies for Food Risk Assessment

Computer-Aided Technologies for Food Risk Assessment

Computer-Aided Technologies for Food Risk Assessment

FOODQUALITY

FOODFRAUD

FOODSAFETY

FOODDEFENSE

EconomicalGain

HealthHarm

Unitentional Intentional

Outline

Traceability SpeciesSubstitution

Dairy Product Fish

Heuristic Molecular

Dairy Farming Analysis“From farm

to Fork”

Food Traceability

Qualitative and

Quantitative certification

STR-DNA Pool Analisys

Heuristic Molecular

Dairy Farming Analysis“From farm

to Fork”

Get around the pool problem

1. Sample Collection (DNA)

2. STRs selection (20)

3. Genotyping Process (STR)

4. Data extraction (RFU)

2 farms for 12 monthsData

Cow → 2 alleles

Pool → vector of alleles

∑ ≈ i = 0

N

i

On-site contamination

& some possible loss

Some loss during

ripening

Assumption

Correct

CorrectMedium

Forgery

CorrectMedium

ForgeryHigh

Forgery

COWs

POOL

DATA

Predicted

POOL

P1 & P2 SSE

SCORE

CMA-ES

W

HEURISTICSIMULATION

Every dairy product

(i.e. 46 pools) has been

simulated 24 times

Heuristic Analysis

0-50-100 % forgery

DATA

COWs

POOL

DATA

Predicted

POOL

P1 & P2 SSE

SCORE

CMA-ES

W

HEURISTICSIMULATION

Covariance Matrix Adaptation

Evolution Strategy

CMA-ES

COWs

POOL

DATA

Predicted

POOL

P1 & P2 SSE

SCORE

CMA-ES

W

HEURISTICSIMULATION

W

∑ ≈

i = 0

N

iw

i

COWs

POOL

DATA

Predicted

POOL

P1 & P2 SSE

SCORE

CMA-ES

W

HEURISTICSIMULATION

W

lower boundary=0.5

m

upper boundary=max ( 3

m,1)

COWs

POOL

DATA

Predicted

POOL

P1 & P2 SSE

SCORE

CMA-ES

W

HEURISTICSIMULATION

Predicted POOLPredicted POOL

0

2000

4000

6000

8000

10000

12000CORRECT POOL

0

1000

2000

3000

4000

5000

6000

7000 ALTERATED POOL

COWs

POOL

DATA

Predicted

POOL

P1 & P2 SSE

SCORE

CMA-ES

W

HEURISTICSIMULATION

P1 - forgeryRate of alleles that are included in the pool’s profile but not in the cows‘

P2 - loss by ripeningPercentage of alleles within the cows’ profile but not detected in the pool

COWs

POOL

DATA

Predicted

POOL

P1 & P2 SSE

SCORE

CMA-ES

W

HEURISTICSIMULATION

SSESum of Squared Errors between

original and predicted pool

SCORE=

P1 · P2 · SSE

Results

Results grouped by pool’s type

Final Results & Conclusion

Vision and Learning

in Fish Species Identification“OK il Pesce

è Giusto”

Fish SpeciesSubstitution

Introduction

Introduction

VISUAL FEATURESVISUAL FEATURES

Introduction

VISUAL FEATURESVISUAL FEATURES

FEATURE ENGINEERING

Introduction

VISUAL FEATURESVISUAL FEATURES

FEATURE ENGINEERING

CLASSIFICATION MODEL

Feature engineering turn your inputs into things the algorithm can understand

DATA Raw Image

DATA

Segmentation

DATA

12 Key-Points

DATA

INFORMATION 30 features

«Is really this the species declaredon the label or not?»

INFORMATION KNOWLEDGE

COMMON NAME LATIN NAME N°European Anchovy Engraulis encrasicolus 125European Pilchard Sardina pilchardus 107Common Pandora Pagellus erythrinus 20Atlantic Mackerel Scomber scombrus 18Gilt-Head Bream Sparus aurata 22European Hake Merluccius merluccius 19Striped Red Mullet Mullus surmuletus 28

Dataset

Classification Model

339 fishes

ANNs: One-Class & Multi-Class Classifiers

Results

Leave One Out Cross and in-field validation

OCC 100% Acc.MCC 100% Acc (only with sardine and anchovy)

Engraulis encrasicolusSardina pilchardusPagellus erythrinusScomber scombrusSparus aurataMerluccius merlucciusMullus surmuletus

MCC

OCC

RAI1 - Live Demo

RAI1 - Live Demo

Limitations

New order/family/genre/species of fish may require

new features

Segmentation will fail if background is not uniform,

and key-points and features too

The two step for key-points should be improved

Key-points interaction should be avoided

FEATURES CLASSIFIEROUTPUT

FEATURES CLASSIFIEROUTPUT

? ? ?OUTPUT

FEATURES CLASSIFIEROUTPUT

DEEP LEARNINGOUTPUT

F.I.S.HUBFish Identification Software Hub

Dat

abas

e

Class

ifier

Valid

ation

Mob

ile A

pp

Fillet

s Rec

ognitio

n

GUIDELINES

> 7k pictures

UK & Italy

25 species

Database

ORDER FAMILY GENUS SPECIES

Clupeiformes Clupeidae Clupea harengus

Clupeiformes Clupeidae Sardina pilchardus

Clupeiformes Clupeidae Sprattus sprattus

Clupeiformes Engraulidae Engraulis encrasicolus

Gadiformes Gadidae Gadus morhua

Gadiformes Gadidae Melanogrammus aeglefinus

Gadiformes Gadidae Merlangius merlangus

Gadiformes Gadidae Pollachius virens

Gadiformes Merluccidae Merluccius merluccius

Perciformes Sparidae Dentex dentex

Perciformes Sparidae Dentex gibbosus

Perciformes Sparidae Diplodus annularis

Perciformes Sparidae Pagellus acarne

Perciformes Sparidae Pagellus bogaraveo

Perciformes Sparidae Pagellus erythrinus

Perciformes Sparidae Pagrus caeruleostictus

Perciformes Sparidae Pagrus pagrus

Pleuronectiformes Pleuronectidae Hippoglossus hippoglossus

Pleuronectiformes Pleuronectidae Limanda limanda

Pleuronectiformes Pleuronectidae Microstomus kitt

Pleuronectiformes Pleuronectidae Pleuronectes platessa

Pleuronectiformes Pleuronectidae Reinhardtius hippoglossoides

Pleuronectiformes Scophthalmidae Psetta maxima

Pleuronectiformes Scophthalmidae Scophthalmus rhombus

Pleuronectiformes Soleidae Solea vulgaris

Database

ORDER FAMILY GENUS SPECIES

Clupeiformes Clupeidae Clupea harengus

Clupeiformes Clupeidae Sardina pilchardus

Clupeiformes Clupeidae Sprattus sprattus

Clupeiformes Engraulidae Engraulis encrasicolus

Gadiformes Gadidae Gadus morhua

Gadiformes Gadidae Melanogrammus aeglefinus

Gadiformes Gadidae Merlangius merlangus

Gadiformes Gadidae Pollachius virens

Gadiformes Merluccidae Merluccius merluccius

Perciformes Sparidae Dentex dentex

Perciformes Sparidae Dentex gibbosus

Perciformes Sparidae Diplodus annularis

Perciformes Sparidae Pagellus acarne

Perciformes Sparidae Pagellus bogaraveo

Perciformes Sparidae Pagellus erythrinus

Perciformes Sparidae Pagrus caeruleostictus

Perciformes Sparidae Pagrus pagrus

Pleuronectiformes Pleuronectidae Hippoglossus hippoglossus

Pleuronectiformes Pleuronectidae Limanda limanda

Pleuronectiformes Pleuronectidae Microstomus kitt

Pleuronectiformes Pleuronectidae Pleuronectes platessa

Pleuronectiformes Pleuronectidae Reinhardtius hippoglossoides

Pleuronectiformes Scophthalmidae Psetta maxima

Pleuronectiformes Scophthalmidae Scophthalmus rhombus

Pleuronectiformes Soleidae Solea vulgaris

Database

FLATFISH

NORMALFISH

ORDER FAMILY GENUS SPECIES

Clupeiformes Clupeidae Clupea harengus

Clupeiformes Clupeidae Sardina pilchardus

Clupeiformes Clupeidae Sprattus sprattus

Clupeiformes Engraulidae Engraulis encrasicolus

Gadiformes Gadidae Gadus morhua

Gadiformes Gadidae Melanogrammus aeglefinus

Gadiformes Gadidae Merlangius merlangus

Gadiformes Gadidae Pollachius virens

Gadiformes Merluccidae Merluccius merluccius

Perciformes Sparidae Dentex dentex

Perciformes Sparidae Dentex gibbosus

Perciformes Sparidae Diplodus annularis

Perciformes Sparidae Pagellus acarne

Perciformes Sparidae Pagellus bogaraveo

Perciformes Sparidae Pagellus erythrinus

Perciformes Sparidae Pagrus caeruleostictus

Perciformes Sparidae Pagrus pagrus

Pleuronectiformes Pleuronectidae Hippoglossus hippoglossus

Pleuronectiformes Pleuronectidae Limanda limanda

Pleuronectiformes Pleuronectidae Microstomus kitt

Pleuronectiformes Pleuronectidae Pleuronectes platessa

Pleuronectiformes Pleuronectidae Reinhardtius hippoglossoides

Pleuronectiformes Scophthalmidae Psetta maxima

Pleuronectiformes Scophthalmidae Scophthalmus rhombus

Pleuronectiformes Soleidae Solea vulgaris

Database

MANY

FEW

Classifier

Examples

Melanogrammus aeglefinus

m=0,052

Psetta maxima

m=0,015

m is the metricgood

Engraulius encrasicolus

m= 0,023

Examples m is the metricwrong

Sardina pilchardus

Sprattus sprattus

m= 1,129

Hippoglossus hippoglossus

Microstomus kitt

m=1,612

Merlangius merlangus

Pollachius virens

m= 1,741

Classifier - Results

t-sne plot for species clusteringFeatures embedding in 10 dimension

t-sne plot for species clusteringFeatures embedding in 10 dimension

Classifier - Results

t-sne plot for species clusteringFeatures embedding in 10 dimension

Classifier - Results

KNN

Mobile Application & Live Demo Record

FISHUB Database

Global Acc. 94%

e.g. Dentex dentex Acc. 86%

Diplodus annularis Acc. 96%

Sardina pilchardus Acc. 96%

The Species Acc. is higher when

the number of picture per species

are elevated and vice-versa

IN FIELD

with DNA analysis

69 fish sampled with pictures and

DNA barcode

9 fraud found by DNA

5 fraud discovered by App

62 correct identification by App

Validation

RESULTS+

+

FILLET

Normal image

Microscope

Toluidine

blue stain

Molecular

sensor

Legend

Fillets Recognition EXPLORATORY TASK

++

The Near-Infrared spectroscopic is a method that

makes use of the electromagnetic spectrum from about

700 nm to 2500 nm. It can penetrate tissues and it is

useful to probe bulk material with essentially no

preparation

Acquisition Visualization Data Model

NIR - SCiO

METHOD DetailsGlobal Accuracy

%

1 normal img 59.8

2 microscope img 49.5

3 normal img + staining 56.6

4 microscope img + staining 53.2

5 SCiO NIR 98.3

+

+

For the POC we selected 10 fish fillets for each considered species:

Solea solea, Pleuronectes platessa, Pangasianodon hypophthalmus.

For methods 1-2-3-4 the pictures were collected and processed with image feature extraction

techniques such as: Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Feature

(SURF) and Gray Level Co-Occurrence Matrix (GLCM). Eventually a SVM classifier was used.

3 random scans per fillets

40 fillets per species

• Gadus morhua• Pleuronectes platessa• Pollachius virens• Epinephelus costae• Synaptura cadenati• Sebastes norvegicus• Merluccius merluccius• Scomber scombrus

RESULTS

98% Accuracy

Latest News

Conclusion

Computer Science

Bio* Data

Model

Interdisciplinarity

PhD List of Publications

1 Journal Paper *

3 Conference Proceeding *

3 Abstract

* first author

Computer-Aided

Technologies for

Food Risk

Assessment

Candidate: Francesco Rossi

Tutor: Prof. Alfredo Benso

XXX Cycle PhD in Computer and Control Eng.

at Politecnico di Torino

N° Name 15 STR 10 STR

1 AGLA29 ok ok

2 MB025 ok ok

3 Z27077 ok ok

4 BMS2142

5 MB071 ok ok

6 SRC276

7 MB064

8 BM1706 ok ok

9 HUJ625 ok

10 BMC1207 ok

11 AGLA232 ok

12 BM3507

13 BM4602

14 BMC6020 ok ok

15 INRA133 ok ok

16 BMS0607 ok ok

17 BMC4214 ok ok

18 BM720 ok

19 BMS4044 ok ok

20 BM0143 ok

GTX 580 DGX-2

600 W

1.5 GB

400€

10 kW

512 GB

420 k€

ProcessedAssumes Beer-Lambert model is valid, and transforms the measured signal to be linear with concentration by doing a log transform and adjusting the result for noise and deviations from the model.

NormalizedPerforms normalization of the signal. This is meant to compensate for changing measurement conditions (e.g. varied scanning distances) that typically occur from sample to sample. Y axis still means reflectance but in normalized units instead of raw reflectance.

Processed and NormalizedFirst assumes Beer-Lambert model (Processed) and then normalizes the results to compensate for differences in the optical path between samples. This is useful, for example, when there is variation in the thickness of the samples.

(log(R))” and NormalizedSimilar to Processed and Normalized, uses a more aggressive form of Processed. Adds more noise, but in some cases may be the only way to create a good model.