04 sem cert hparticipation_final
TRANSCRIPT
1Information Technologies InstituteCentre for Research and Technology Hellas
CERTH at MediaEval 2014 Synchronization of Multi-User
Event Media Task
Konstantinos Apostolidis, Christina Papagiannopoulou,
Vasileios Mezaris
Information Technologies Institute / Centre for Research and Technology Hellas
2Information Technologies InstituteCentre for Research and Technology Hellas
• Media from different users related to an event are exchanged/shared in a number of ways.
– time information attached to some of the captured media may be wrong
– geolocation information may be missing
Objective: The alignment and presentation of media of different users in a consistent way, preserving the temporal evolution of the event.
Introduction
4Information Technologies InstituteCentre for Research and Technology Hellas
Time Synchronization (1/4)
• Near Duplicate image detection– SIFT descriptors extraction.– Clustering to form a vocabulary.– Encoding of each image using VLAD encoding.– Keypoints geometry consistency checking using Geometric Coding to
refine the returned near duplicate images.• Modified Near Duplicate image detection
– Use color information (HSV histograms). Near duplicate candidates that are very similar in color are considered even if the Geometric Coding score is relatively low.
5Information Technologies InstituteCentre for Research and Technology Hellas
Time Synchronization (2/4)
• Apply the modified NDD on the union of all galleries to come up with pairs of near duplicate images belonging to different images.
• Filter results, rejecting pairs of which:– the location distance of the two photos is greater than a distance threshold.– the time difference between the photos is above a time threshold.
• The remaining of near duplicate photos belonging to different galleries are considered as links between those galleries.
6Information Technologies InstituteCentre for Research and Technology Hellas
Time Synchronization (3/4)
• Graph Construction– Galleries → Nodes– Links between galleries → Edges – Number of links between galleries → Edge weights
7Information Technologies InstituteCentre for Research and Technology Hellas
Time Synchronization (4/4)• Graph traversal:
1. Start from the node corresponding to the reference gallery.2. Select the edge with the highest weight.3. Compute the time offset of the node on the other end.
Time offset is equal to the median of time differences of near duplicate image pairs from the corresponding galleries.
4. Repeat steps 2 and 3 considering as possible starting point any visited node.
• Once the galleries time offsets are calculated, align the image timestamps.
8Information Technologies InstituteCentre for Research and Technology Hellas
Media Clustering to Events
• Cluster events using the corrected timestamps.
• Two approaches are followed:1. Consider all photo galleries as a single photo collection
and perform clustering in this collection.2. Make pre-clustering within each gallery separately and
then merge the clusters across the galleries.
9Information Technologies InstituteCentre for Research and Technology Hellas
1st Clustering Approach (1/3)• Sort all the photos according to the corrected timestamps• Construct the similarity matrix using the similarity measure:
where ti, tj the timestamps of the images i,j and K is a parameter for the sensitivity of the measure.
• The larger the value of K the coarser the clusters created.
K = 30 min K = 12 h
10Information Technologies InstituteCentre for Research and Technology Hellas
• Compute the novelty scores for each photo i, vK, to detect the cluster boundaries– Correlate a checkerboard kernel g along the main diagonal of each SK. Kernel g
– , where l = 1
• Detect the peaks that have difference greater than a threshold.
K = 30 min K = 12 h
Repeat the above procedure for K = 15, 30, 60, 120, 720, 1440 min
1st Clustering Approach (2/3)
11Information Technologies InstituteCentre for Research and Technology Hellas
1st Clustering Approach (3/3)• Select the boundary list, BK = {b1, … , bnK}, which forms the larger within-
cluster similarity and the lower between-cluster similarity, C(BK), using :
• The time alignment may not be perfect.• Split each of the clusters using the geolocation information.
– Photos with close geolocation are assigned to the same cluster– Photos which have great geolocation difference are assigned to
different clusters• The photos that do not have geolocation information are assigned to the
geo-cluster which is more similar according to the color information (e.g. HSV histogram).
12Information Technologies InstituteCentre for Research and Technology Hellas
2nd Clustering Approach (1/2)• In each gallery, find the minimum time difference of dissimilar
photos which is greater than the maximum time difference of the near-duplicate photos (based on the similarity matrix of GC).
13Information Technologies InstituteCentre for Research and Technology Hellas
2nd Clustering Approach (2/2)• The clusters that are formed by these time gaps, are merged
according to time and geolocation similarity.
• For the clusters that do not have geolocation information, the merging is continued by considering:– the time and low-level feature similarity – the time and the concept detector confidence similarity scores
14Information Technologies InstituteCentre for Research and Technology Hellas
Submitted runs• Run #1: Compute gallery time offsets using our modified NDD. Concept
Detection scores are used to merge clusters using the 2nd approach in clustering.
• Run #2: Compute gallery time offsets using our modified NDD. HSV histogram similarity is used to merge clusters using 2nd clustering approach.
• Run #3: Compute gallery time offsets using our modified NDD. Clustering is performed using the 1st clustering approach.
• Run #4: Compute gallery time offsets using our modified NDD and traverse of the graph by considering the weights of multiple edges. Concept Detection scores are used to merge clusters using the 2nd clustering approach.
• Run #5: Compute gallery time offsets using NDD without HSV color information. Concept Detection scores are used to merge certain events using the 2nd clustering approach.
15Information Technologies InstituteCentre for Research and Technology Hellas
Evaluation (1/2)
• Time synchronization:– Modified NDD approach gives the best results in time
alignment for the Vancouver testset.– Standard NDD yields a slightly better time synchronization
than modified NDD, for the London testset.
Run #1 Run #2 Run #3 Run #4 Run #5
Vancouvertestset
Time Sync. Precision 0.9118 0.9118 0.9118 0.5294 0.9118
Time Sync. Accuracy 0.7375 0.7375 0.7375 0.7014 0.7279
Run #1 Run #2 Run #3 Run #4 Run #5
Londontestset
Time Sync. Precision 0.6111 0.6111 0.6111 0.2222 0.6389
Time Sync. Accuracy 0.7127 0.7127 0.7127 0.6996 0.7299
16Information Technologies InstituteCentre for Research and Technology Hellas
Evaluation (2/2)• Sub-event Clustering:
– Exploitation of consistent timestamps in a gallery and the use of Concept Detection scores gives a good performance for the Vancouver testset.
– HSV histogram similarity clearly gives the best clustering results for the London testset.
Run #1 Run #2 Run #3 Run #4 Run #5
Vancouvertestset
Clustering Rand Index 0.9770 0.9734 0.9526 0.9601 0.9656
Clustering Jaccard Index 0.2581 0.2315 0.2856 0.1782 0.2861
Clustering F-Measure 0.2052 0.1880 0.2222 0.1512 0.2225
Run #1 Run #2 Run #3 Run #4 Run #5
Londontestset
Clustering Rand Index 0.9885 0.9910 0.9838 0.9829 0.9863
Clustering Jaccard Index 0.5051 0.5614 0.3232 0.2739 0.4849
Clustering F-Measure 0.3356 0.3596 0.2443 0.2150 0.3266
17Information Technologies InstituteCentre for Research and Technology Hellas
Conclusions• Using color information in modified NDD, we can discover
more pairs of images between different galleries.• However, due to multiple events being carried out at the
same location with characteristic color background in London testset, modified NDD returned some wrong pairs, yielding to worse results than the standard method.
• In sub-event clustering, the use of HSV histograms similarity seems to give a better performance than the Concept Detector confidence scores, in a dataset with increased color diversity, such as London testset.
18Information Technologies InstituteCentre for Research and Technology Hellas
Questions?
More information:http://www.iti.gr/~bmezaris [email protected]