the fish4knowledge project disclosing computer vision errors to end-users emma beauxis-aussalet,...
TRANSCRIPT
The Fish4Knowledge Project
Disclosing Computer Vision Errors to End-Users
Emma Beauxis-Aussalet, Lynda Hardman, Jacco Van Ossenbruggen, Jiyin He,Elvira Arslanova, Tiziano Perrucci
12 December 2014 CWI Scientific Meeting 1
Monitoring Fish Population
2
We count fishto study ecosystems
Monitoring Fish Population
3
Why use Computer Vision?• It can count fish from each species• It supports short- to long-term research• It is not intrusive, and cost-effective
Monitoring Fish Population
4
But… new practices need to be introduced,and scientific validity needs to be assessed.
Why use Computer Vision?• It can count fish from each species• It supports short- to long-term research• It is not intrusive, and cost-effective
Detecting Fish
5
Collect examples of fish(Ground-Truth)
6
Collect examples of fish(Ground-Truth)
Construct fish models
Detecting Fish
7
Collect examples of fish(Ground-Truth)
Construct fish models
Classify fish species as the most similar model
Detecting Fish
Motivations for HCI Research
8
Support uncertainty-aware data analysis • What are the uncertainty factors?• How to inform ecologists about each factor?• How to support user assessment of end-results?
Here the Octopus appeared.
(½Φ )-(π√⅞)
How precise is this?
12 December 2014 CWI Scientific Meeting
What are the uncertainty factors?
9
Interactions of Uncertainty Factors
10
Computer Vision Errors
Interactions of Uncertainty Factors
11
Some fish are misclassified
Computer Vision Errors
Interactions of Uncertainty Factors
12
Poor ground-truth yields poor models
Ground-Truth Quality
Computer Vision Errors
Interactions of Uncertainty Factors
13
Poor imagesyield more errors
Ground-Truth Quality
Computer Vision Errors
Image Quality
Interactions of Uncertainty Factors
14
Typhoons yield poor images?What confidence intervals?
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
15
Missing videos?
Number of Video Samples
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
16
Some species swim in & out the field of view
Number of Video Samples
Duplicated Individuals
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
17
Fields of view target specific species
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
18
Fields of view target specific species
and shift overtimeNumber of Video
Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Interactions of Uncertainty Factors
19
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Computer Vision Errors
Biases & Noisein Specific Output
Image Quality
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Biases & Noisein Specific Output
Image Quality
Conveying Uncertainty Factors
20
Confusion Matrices
Computer Vision Errors
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Image Quality
Conveying Uncertainty Factors
21
Confusion Matrices
Computer Vision Errors
Biases & Noisein Specific Output
LogisticRegression
12 December 2014 CWI Scientific Meeting
Conveying Computer Vision Errors with Confusion Matrices
22
Here the Octopus appeared.
(½Φ )-(π√⅞)
How precise is this?
12 December 2014 CWI Scientific Meeting
Conveying Computer Vision Errors with Confusion Matrices
23
Here the Octopus appeared.
(½Φ )-(π√⅞)
How precise is this?
Without torture, no science.
Russian Proverb
State-of-the-Art
24
Confusion Matrix
State-of-the-Art
25
TypicalVisualization
State-of-the-Art
26
State-of-the-Art
27
State-of-the-Art
28
Diagonals are correct fish (TP).The rest are errors.
State-of-the-Art
29
Columns are missed fish.(FN)
State-of-the-Art
30
Rows are added fish.(FP)
State-of-the-Art
31
Rows & Columns are cumulated
State-of-the-Art
32
Advancedmeasurementsare repeated
Proposed Metric & Visualization
33
• 3 basic concepts: correct, added, missed fish• 2 Main sources of errors• Number & Proportions of error • Simple metric for extrapolations
Proposed Metric & Visualization
34
• 3 basic concepts: correct, added, missed fish• 2 Main sources of errors• Number & Proportions of error • Simple metric for extrapolations
Proposed Metric & Visualization
Numbers Proportions
Proposed Metric & Visualization
Numbers Proportions
ImproveGround-Truth?
Proposed Metric & Visualization
Numbers Proportions
Improvealgorithm?
ImproveGround-Truth?
38
Issues Tackled
39
Issues Tackled
Large number of TNconceals uncertainty
40
Issues Tackled
Large number of TNconceals uncertainty
Information is lostabout errors interdependence
41
Issues Tackled
Information is lostabout errors interdependence
FP for one class are FN for another
Large number of TNconceals uncertainty
42
Issues Tackled
Information is lostabout errors interdependence
Large number of TNconceals uncertainty
Class proportions can vary
43
Issues Tackled
Information is lostabout errors interdependence
Large number of TNconceals uncertainty
Class proportions can vary
Using one single type of curvecan hide differences
12 December 2014 CWI Scientific Meeting
ConveyingComputer Vision Biaseswith Logistic Regression
44
in collaboration with Bas Boom
Without torture, no science.
Russian Proverb
Logistic Regression Method
45
Logistic Regression Method
46
Logistic Regression Method
47
Logistic Regression Method
48
Logistic Regression Method
49
Logistic Regression Method
50
Visualization of Errors
51
Detail of the chances of biases
Achievements & LimitationsBefore After
Achievements & Limitations
Fish counts are improved
Before After
Achievements & LimitationsBefore After
Fish counts are improved
Achievements & LimitationsBefore After
Fish counts are improved
Achievements & Limitations
Biases arereduced
Before After
Fish counts are improved
Achievements & Limitations
But beware further variations of class proportions!
Before After
Biases arereduced
Fish counts are improved
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Image Quality
Future Work
58
Computer Vision Errors
Biases & Noisein Specific Output
Integrate Computer Vision errorsinto ecologists’ statistical framework
Number of Video Samples
Duplicated Individuals
Field of View
Ground-Truth Quality
Image Quality
Future Work
59
Computer Vision Errors
Biases & Noisein Specific Output
Integrate Computer Vision errorsinto ecologists’ statistical framework
User studies with our visualizations
Number of Video Samples
Ground-Truth Quality
Image Quality
Future Work
60
Integrate Computer Vision errorsinto ecologists’ statistical framework
Computer Vision Errors
Biases & Noisein Specific Output
User studies with our visualizations
Duplicated Individuals
Field of View
Develop measurement methods
