© 2010 microsoft corporation. all rights reserved. quality assurance: towards tools for...

© 2010 Microsoft Corporation. All rights reserved.

Quality Assurance: Towards Tools for Characterizing and

Comparing Digital Documents

Natasa Milic-FraylingMicrosoft Research Cambridge UK

www.planets-project.eu


What is the problem?

• Digital is a victim of its own success

i.e., the advances in digital technologies that make digital media broadly used and adopted

Document formats, software and hardware are becoming obsolete faster than we can ensure the forward compatibility of the content.


What are technical solutions?

• We have two main strategies:

emulation and simulation– Create emulators of hardware

and simulators of software systems to enable old programmes to run and old data to be used.

content migration– Migration to

standards that are likely to be supported in the future.


Preservation and

Long-term

Access through

NETworked

Services

• Ensure long-term access to Europe’s cultural and scientific heritage• Improve decision-making about long

term preservation• Ensure long-term access to valued

digital content• Control the costs through automation,

scalable infrastructure• Ensure wide adoption across the user

community• Establish market place for preservation

services and tools

• Build practical solutions• Integrate existing expertise, designs

and tools• Share and build


The British Library

National Library, Netherlands

Austrian National Library

State and University Library, Denmark

Royal Library, Denmark

National Archives, UK

Swiss Federal Archives

National Archives, Netherlands

Hatii at University of Glasgow

University of Freiburg

Technical University of Vienna

University at Cologne

Tessella Plc

IBM Netherlands

Microsoft Research, Cambridge

ARC Seibersdorf research

PLANETS Partners

http://www.bl.uk/

http://www.bl.uk/

http://www.bl.uk/

http://www.statsbiblioteket.dk/

http://www.statsbiblioteket.dk/

http://www.en.nationaalarchief.nl/default.asp



http://www.ibm.com/nl/

http://research.microsoft.com/

http://research.microsoft.com/


PLANETS Sub Projects


CONVERSION TOOLSpreserving office documents


Microsoft & PLANETS: Preserving Office Documents• Microsoft Research role within PLANETS:

– Conversion of binary Microsoft Office Documents into Office Open XML File Format (OpenXML)

• We extended the effort to include other formats– More legacy formats, e.g. WordPerfect– Other open standards, e.g. Open Document Format.

Binary MS Office OpenXML

WordPerfect ODF

Binary MS Office OpenXML

DOS Word UOF


Document Conversion Tools – Our Approach

• Three-step approach, resulting in a modular and extendible infrastructure– Identify existing conversion tools and libraries– Wrap these tools and libraries into re-usable components– Integrate these components into PLANETS and other

systems.

• If possible, do not use the office applications (e.g., Microsoft Office or OpenOffice.org)– They are designed as interactive applications– Message boxes might pop up (“Do you want …”)– Unclear license question when running on a server.


Reusable Components

Transformer Box (Wrapper)

“Binary OpenXML”TB

Interface

Watch Folder Tool

Web Service

ToooXML (GUI)


Extendible Architecture


“ODF OpenXML”


“WP OpenXML”


“Binary OpenXML”TB

Interface

Watch Folder Tool

Web Service

ToooXML (GUI)


More Technical Details (1)

• Currently two types of wrappers for– Command-line tools (stand-alone executables)• OpenXML/ODF Translator (OpenXML ODF)• OpenXML Document Viewer (OpenXML HTML)

– Microsoft conversion libraries (CNV libraries)• WordPerfect RTF• RTF OpenXML• …

• We allow wrappers to be chained– WordPerfect RTF OpenXML ODF.


Microsoft Word

More Technical Details (2)• Microsoft conversion libraries (CNV libraries)– Originally designed to import/export “foreign”

document formats into/from Microsoft Word– Based on the Microsoft Conversion API• Foreign2RTF• RTF2Foreign

– Transformer Box CNV Wrapper follows this API.

Transformer Box CNV Wrapper

CNV LibraryRTF2Foreign Foreign2RTF


Supported Formats Source formats

WordPerfect 5 WordPerfect 6 DOS Word Word 2, 6, 95 Word 97-2003 RTF ODF OpenXML

Target formats OpenXML ODF UOF HTML XCDL (format defined in

PLANETS/PC)


CONVERSION SERVICESpreserving office documents


Conversion applications and service


SIMILARITY ASSESSMENTunderstanding the quality criteria


How do we explore and compare digital artefacts

• Perceptive aspects of the digital object– In the past printed version of the document and

screen display

• Interactive aspects of the digital objects– Dynamic content includes both individual artefacts

and the `stream characteristics‘.

• Non-perceptive aspects of the digital objects– Document object model, cashed data, action

generated metadata, hidden formulas, etc.


EXAMPLE: Perceptive features for Word Documents

• Two objects in different formats are mapped onto the normalize form– E.g., a WP file converted into .docx. For both we

create an XPS representation of the document

• Feature extraction and comparison– For each feature develop a `digital object probe‘ that

extract the feaeture and measure a property of the feature

– E.g., pass XPS through OCR package and extract various layout features.


Conversion applications and service


What is ahead of us?• Research

– What is the relationship between the human criteria and automated measurements? What usage scenarios do we aim for?

• Technology– What ‘instruments’ do we need to extract and measure properties

of the digital content? – How do we automate the process of inspection and quality

assurance?• Legal

– How do we run legacy software as services? We need updated licensing agreements.

– How to provide services that combine open source and non-open source software?


THANK YOUContact: Natasa Milic-FraylingMicrosoft Research [email protected]

mailto:[email protected]