cadal digital library wu jiang-qin,zhuang yue-ting pan yun-he college of computer science, zhejiang...
TRANSCRIPT
CADAL Digital Library
Wu Jiang-Qin,Zhuang Yue-Ting Pan Yun-he
College of Computer Science, Zhejiang University,China
November 18,2006 November 18,2006
The 2nd International ConferenceThe 2nd International Conferenceon Universal Digital Libraryon Universal Digital Library
(ICUDL 2006) (ICUDL 2006)
Outline
1111
4444
2222
5555
Conclusion and Future Work
Introduction
Unified Paralleling Search
Multimedia Analysis and Retrieval
6666
3333
Bilingual services
Chinese Calligraphy Character Retrieval
Outline
1111
4444
2222
5555
Conclusion and Future Work
Introduction
Unified Paralleling Search
Multimedia Analysis and Retrieval
6666
3333
Bilingual services
Chinese Calligraphy Character Retrieval
CADAL The China-Us Million Book Digital Library(CAD
AL) is an international cooperation program between China and the US.
The objective of CADAL project , is to create a free-to-read, searchable collection of one million book, available to everyone over the internet.
CADAL is the important part of Universal Digital Library(UDL), universal access to human knowledge.
The challenges and services (1)
the amount of the digital resources including digital books and multimedia for research and education can reach 100 terabyte(The number of digital books is 1,023,425 by October of 2006,including previous Chinese ancient books, Chinese minguo books ,Chinese Modern books, Chinese degree dissertation,English books,image,video etc..
active services of unified paralleling search for the different types of digital resources
The challenges and services (2)
image, video,3-D model and other types of media resources, various types of media resources are included in the CADAL resources.
the services of quickly retrieving and structurally browsing of multimedia documents including image, video
The challenges and services (3)
there are two kinds of language digital books. Chinese and English, in the CADAL resources.
the services of bilingual translation
The challenges and services (4)
traditional Chinese culture resources are important part of the CADAL resources.
the services related to Chinese traditional culture resources.
Outline
1111
4444
2222
5555
Conclusion and Future Work
Introduction
Unified Paralleling Search
Multimedia Analysis and Retrieval
6666
3333
Bilingual services
Chinese Calligraphy Character Retrieval
Background
TB volume of various types of digital resources, such as dissertation, ancient minguo book, modern book, minguo journal, English book, drawing, video and illustration are available in the CADAL, which is one of the distinct characteristic of CADAL. So CADAL presents a challenge for the technique of searching resources based on metadata.
Metadata
Dublin core metadata is used to describe the million digital books in the CADAL project. Metadata corresponding to the other types of multimedia resources are used to describe them. Independent data map is designed for each kind of resource metadata.
Unified parallel searching In order to meet the requirements
of different users and improve the user’s interactive experience, the service for the different types of digital resources is provided for users’ convenient searching.
Outline
1111
4444
2222
5555
Conclusion and Future Work
Introduction
Unified Paralleling Search
Multimedia Analysis and Retrieval
6666
3333
Bilingual services
Chinese Calligraphy Character Retrieval
Background As the digital library contains unstructured
multimedia resources such as images, videos, audios etc besides digital books, effective and efficient analysis and retrieval of multimedia resources is a challenging problem in the CADAL digital library.
Here we examine the analysis and retrieval issues related to two primary kinds of multimedia, image and video.
Contents Content-based Image Retrieval Image retrieval by peer indexing Image annotation Image search engine Video analysis system Video Browser(structure and summary) Metadata-based Video Retrieval
Content based image retrieval
Extracting visual features color feature： color histogram, color moment, col
or coherence vector, color correlgram texture： Tamura textural feature and co-occurren
ce textural feature
relevance feedback Make image retrieval coincide with user’s r
equirement
Content based image retrieval
Positive example
Negative example
Query example
Image searching
Relevance feedback
Image retrieval by peer index A new scheme for image indexing, Peer Index,
is the method that describe images through semantically relevant peer images.
In particular, each image is associated with a two-level peer index, including global peer index: describing the “data
characteristics” of this image personal peer indexes: describing the “user
characteristics” of an individual user with respect to this specific image
Both types of peer index are learned interactively and incrementally from user feedback information.
Peerindex-based image retrieval
Semantic query
semantic relevance feedback
Image annotation Automatic semantic annotation for images by
machine learning and statistical modeling Classify the training images, and create a semantic
skeleton for each class of the training image.
Classify new image with Support Vector Machine automatically, and describe it using the semantic skeleton
Select the key words for the image by statistical methods
images
images
tiger
annotation
classify
............
statistical learning
classifysegment
Images
annotate
Visual similar
Image blobs
Semantic skeleton
Image annotation
Text based image retrieval
Query text ： bonsai
Image search engine We implemented an image search engine,
Octopus, which provides Peer Index and relevance feedback to avoid the gap between the semantics and low-level features, according to the intuitive and simple idea that the semantic concept is hidden in each image and the semantic concept appears apparently in the relation between the image and the other images.
Integrating into CADAL DL
imagesBooks scanner
Image Manager
store
images metadata feature
CADAL image repository
CADAL portal
Image retrieval systemuser
images
Other images
WWW
BrowseRetrieveRelevance feedback
The image retrieval interface
Our target for video
Analyze multimodal information, such as the visual, the audial, motion and caption to generate structural information and video summary
Support video browsing and video retrie
val based on metadata and structural information efficiently
Main idea
Nonlinear browsing： Generate structural indexing such as key frame, shot and shot group from the original video stream
Content compression： Analyze time sequence in video stream, eliminate redundant data, and generate the summary and the highlight scene for the original video.
System
Video Fusion Analysis System (VideoFAS) VideoBrowser
Video dataVideo fusion analysis system
video repository
Metadata database
Feature database
CADAL video repositoryVideo browser
user
CADALportal
VideoFAS－ system interfaceOriginal Video
Video shot
Similar Video shots are Clustered together
Basic operation Importing and Saving
Appending
Separating the video stream into video and audio data
transcoding and compressing
VideoFAS－ system functions
Feature Extraction Visual feature
color ： color histogram, color moment, color coherence vector, color correlgram
Texture ： Tamura textural feature and co-occurrence textural feature
shape： contour feature Audial feature
temporal feature： zero-crossing rate Frequency feature ： Mel coefficient、 tone and sub-ba
nd statistical feature Target Feature
Integrate OpenCV face detection module into the system Extract the face features
VideoFAS－ system functions
Video structuring shot detection
Cut shot detection Transition shot detection
key frame extraction
Similar shot grouping group the shots based on Support Vector Machine
VideoFAS－ system functions
Original Video Shot Sequence
Video Shot Clustering
Video Shot Cluster A
Video Shot Cluster B
Video Shot Cluster C
Video Shot Cluster D
Video Shot Cluster E
Video summarization Summarize by Mining Non-Trivial Repeating Patterns
Extract frequent and non-trivial shot sequence to generate video summary
VideoFAS－ system functions
Original Video Shot Sequence
A B C D
A B C D A C D E A B C D
Metadata annotation Annotate Video clip with metadata conform
to Dublin Core Standard
Save the metadata and the video structural information in database
VideoFAS－ system functions
VideoBrowser－ framework
WWWVideo repository
Original video data
Video catalog
Video summaryuser
Content service
VideoBrowser－ system interface
VideoBrowser－ system interface
metadata
Video structural information
media player
video data
annotation structuring
Online storage
Disk array
switcher
Archive server taper （ offline storage ）
Web server
firewall
InternetRetrieval service
Web Movies
System architecture
summarization
Outline
1111
4444
2222
5555
Conclusion and Future Work
Introduction
Unified Paralleling Search
Multimedia Analysis and Retrieval
6666
3333
Bilingual services
Chinese Calligraphy Character Retrieval
Background
As there are both English and Chinese books in CADAL, bilingual services are required for users to access resources in any language.
Services Some technologies and prototypes have been
developed by north technical center on how to carry out the multi-layered bilingual machine translation in English and Chinese books, such as the metadata translation between English and Chinese the accurate translation of proper nouns such as names
for unique individuals, events,or places the selective translation in a full-text context the translation of Old Chinese text the distributed translation service technique.
Services
An online translation service is integrated into the CADAL digital library. Users can be directly conducted semantic-based
multi-linguistics retrieval of required information in our CADAL digital library.
The translation of contents of a page on line. The translation of metadata of a digital book.
Bilingual Search
The translation of contents of a page
Outline
1111
4444
2222
5555
Conclusion and Future Work
Introduction
Unified Paralleling Search
Multimedia Analysis and Retrieval
6666
3333
Bilingual services
Chinese Calligraphy Character Retrieval
Background
Since most people are interested in the art of the beautiful styles of calligraphy character rather than the meaning of the character, the service of Chinese calligraphy character retrieval is provided in the CADAL digital library, treating them just as they are images without recognizing them like OCR does.
Calligraphy art still alive in:
Key issues
Feature extraction: character complexity, stroke density and shape, the three kinds of features of the calligraphy character are proposed
similarity matching cost: retrieve relevant images according to it.
Contents
Chinese Calligraphy Page Segmentation Features Extraction Character Image Retrieval
Chinese Calligraphy Page Segmentation
The page image are binarized with characters in black and the background in white.
Cut the page into columns according to the vertical projecting histogram, and columns continued to be cut into individual characters.
All the characters are normalized in order to keep scale invariant Contour information,
Page segmentation
Features Extraction
shape character complexity
shape representation Calligraphic character’s shape is represented
by its contour points. The polar coordinates is used to describe
directional relationship of points instead of the Cartesian coordinates.
For direction, we use 8 bins in equal degree size to divide the whole space into 8 directions.
For radius, we use 4 bins For each point of a given point set composed of
sampling points, its approximate shape context is described by its relationship with the remaining points in weighted bins.
shape representation
2630 w
Contour point
)}(:{#)( kbinqpqkW jiji
2424 w
Calligraphy Character Complexity We use Calligraphy Character Complexity as a
filter at the beginning to discard the calligraphy character that has no possibility to be similar to the query.
iL
L1
L L be the number of sampled contour points from the be the number of sampled contour points from the query and query and LLi i be the number of sampled contour points be the number of sampled contour points from candidate image.from candidate image. is the threshold obtained by is the threshold obtained by experience. experience.
Character Image Retrieval Compute the values of the character complexity of the
calligraphy character. Normalize the scale size of the query and sample its
contour points. Filter the candidate images by character complexity Extract the shape feature and employ the shape
matching method introduced in [6] to compute the matching cost for every remaining candidate image and the query.
Rank the results according to the matching cost, and return.
The calligraphy character retrieval
interface of browsing the original works
Outline
1111
4444
2222
5555
Conclusion and Future Work
Introduction
Unified Paralleling Search
Multimedia Analysis and Retrieval
6666
3333
Bilingual services
Chinese Calligraphy Character Retrieval
All the services have been accessed by the users from over 70 countries 280.000 times per
day.
Conclusion and Future Work
With the increase of the number of the users and the amount of the resources. future work with CADAL digital library will proceed in several directions: Improving the performance of the current
services, to be more complete and be more stable;
Continuing exploring the application of multimedia in Digital Library.
Thanks ！Welcome Visiting
CADAL Digital Library(WWW.CADAL.ZJU.EDU.CN)
Email: Email: [email protected] [email protected] [email protected]@zju.edu.cn