mediaeval 2015 - the placing task at mediaeval 2015
TRANSCRIPT
PLACING TASK 2015
Jaeyoung Choi (ICSI, Berkeley / TU Delft, Netherlands) Claudia Hauff (TU Delft, Netherlands)
Olivier Van Laere (Blueshift Labs, San Francisco) Bart Thomee (Yahoo Labs, San Francisco)
Sept 15th, 2015 Wurzen, Germany
TASK CHANGES
• two new sub-tasks• locale-based sub-task• mobility-based sub-task
• organizer baselines
• live leaderboard
LOCALE-BASED SUBTASK
• given a video or a photo, place it as accurately as possible on a map
LOCALE-BASED SUBTASK
• participants were given a geographic hierarchy of locales across the world• principally based on the GADM administrative boundaries• supplemented with neighborhood data from ClickThatHood
• participants choose the locale in the hierarchy where they most confidently believe a photo/video was taken
• or, if sufficiently confident, they pick a longitude and latitude coordinate
HIERARCHY DATA EXAMPLE
• United States@CA|Calif.|California@San Francisco@South of Market• ‘@‘ separates each layer in the hierarchy • ‘|’ separates any alternative name spellings
MOBILITY-BASED SUBTASK
• participants were given a sequence of photos captured by a user within a certain city• not all of the photos within the sequence were geo-tagged
• participants were asked to estimate the location of all photos with missing geo-coordinates
TASK DATASET
• drawn from the Yahoo Flickr Creative Commons 100 Million (YFCC100M) dataset
training testing#photos #videos #photos #videos
locale-based 4,672,382 22,767 931,573 18,316
mobility-based 148,349 0 33,026 0
PRECOMPUTED FEATURES• textual metadata
• as included in YFCC100M• visual features
• LIRE, GIST, SIFT• audio features
• MFCC, Pitch (Kaldi, SAcC)
paul bica
ORGANIZER BASELINES
• two open-source baselines were provided, one for each subtask• locale-based baseline - LM-based using textual metadata [Van
Laere et al. 2013] • mobility-based baseline - extrapolates location from temporal
neighbors [Hauff et al. SIGIR 2012]
https://github.com/ovlaere/placing-text/blob/mediaeval2015/README_placing2015.mdhttps://github.com/chauff/ImageLocationEstimation
LIVE LEADERBOARD• participants could submit runs and view their relative
standing toward others• evaluated on a development set, approx. 30% of test set• considered to be useful by participants
TASK EVALUATION
• locale-based subtask• hierarchical distance between ground truth locale and
predicted locale in the place hierarchy• geographic distance between ground truth coordinate and
predicted coordinate • mobility-based subtask
• geographic distance-based metric
• custom formula was used for the hierarchy-based metric• Karney’s formula was used for the distance-based metric
TASK EVALUATION
RUNS• run 1
• only provided textual metadata may be used• run 2
• only provided visual & aural features may be used• run 3
• only provided textual metadata, visual features and the visual & aural features may be used
• run 4–5• everything is allowed, except for crawling the exact items
contained in the test set, or any items by a test user taken within 24 hours before the first and after the last timestamp of a photo sequence in the mobility test set
PARTICIPANTS STATISTICSlocale mobility
run1 run2 run3 run4/5 run1 run2 run3 run4/5
CERTH/CEA LIST O O O O
RECOD O O O O
ImCube O O O O O O O OJKU_
Satellite O O O
Geo_ML O
TUDelft O (late)
LOCALE RESULTS - RUN 1cu
mula
tive
perc
enta
ge
0
15
30
45
60
75
geographic distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCubeRECOD Geo_MLBaseline
cum
ulativ
e pe
rcen
tage
0
15
30
45
60
75
hierarchical distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCubeRECOD Geo_MLBaseline
LOCALE RESULTS - RUN 2cu
mula
tive
perc
enta
ge
0
8
16
24
32
40
geographic distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCubeRECOD TUD*
cum
ulativ
e pe
rcen
tage
0
8
16
24
32
40
hierarchical distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCubeRECOD TUD*
LOCALE RESULTS - RUN 3cu
mula
tive
perc
enta
ge
0
15
30
45
60
75
geographic distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCubeRECOD Baseline
cum
ulativ
e pe
rcen
tage
0
15
30
45
60
75
hierarchical distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCubeRECOD Baseline
LOCALE RESULTS - RUN 4/5cu
mula
tive
perc
enta
ge
0
15
30
45
60
75
geographic distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCubeRECOD Baseline
cum
ulativ
e pe
rcen
tage
0
15
30
45
60
75
hierarchical distance from groundtruth
1m 100m 10km 1000km
CERTH/CEA LIST ImCubeRECOD Baseline
MOBILITY RESULTS - RUN 1+2cu
mula
tive
perc
enta
ge
0
20
40
60
80
100
geographic distance from groundtruth
1m 100m 10km 1000km
ImCube JKU_Satellite
cum
ulativ
e pe
rcen
tage
0
20
40
60
80
100
geographic distance from groundtruth
1m 100m 10km 1000km
ImCube JKU_Satellite
MOBILITY RESULTS - RUN 3+4cu
mula
tive
perc
enta
ge
0
20
40
60
80
100
geographic distance from groundtruth
1m 100m 10km 1000km
ImCube JKU_Satellite
cum
ulativ
e pe
rcen
tage
0
20
40
60
80
100
geographic distance from groundtruth
1m 100m 10km 1000km
ImCube
CONCLUSIONS
• no visible trend; many different approaches• pre-processing, language model, clustering, GP, etc.
• limited time available for participants• leaderboard, features were helpful
• local semantic features from visual cue to be added