Practical Approaches to SharingInformation at Raytheon

Taxonomies, Metadata and Beyond

Presented by

Christine JM. Connors

Christine_Connors@raytheon.com

KMPro / KMForum, Bentley College, Waltham, MA

June 30, 2004

2

Data Discovery – Data Discovery – What we LearnedWhat we Learned• Level of information management varies

• 85% of our information is unstructured

• Over 90% of information is not tagged

• High proportion of tagged documents result of templates, and therefore relay bad data

• Ethan Frome – over 200 documents

• Automatically extracted data imprecise

• “The flight to Dallas is cancelled”

• Greater than 13% of information is exactly duplicated

• “Near” duplication harder to determine but potentially more costly

• Worst duplication in File Servers / Shared Drives

• Difficult to determine true age of document due to web scripting, date of publication to public drive

• Over 23% of sample data not modified in previous 5 years

• Over 68% of sample data not modified in previous 2 years

3

Intranet Search and Browse SurveyIntranet Search and Browse SurveyJune 2003June 2003

• 16 multiple choice questions, 1 optional free-text comment field

• 516 surveys started (clicked into)

• 199 responses over 3 weeks

• 39% completion rate

• 101 comments on “How can we improve the intranet search and browse capabilities?”

• 51% comment rate by survey participants

4

How can we improve the intranet search How can we improve the intranet search and browse capabilities? and browse capabilities?

101 user comments frequently included:

• Qualify searches by function, organization, and business

• Qualify searches by date

• Qualify searches by document type (especially web pages)

• Provide sorting of results by date, document type

• Provide category search

• Do not change URLs of pages (users bookmarked)

• Reduce number of search results

• Google (mentioned 32 times in comments)

5

Results SummaryResults Summary

Search/browse

• About ¼ (26%) of respondents find the current capabilities “Good” or “Excellent”

• About ¼ (24%) of respondents consistently locate helpful information• About ¼ (22%) of respondents indicate they are consistently successful using

keyword searching• About ¼ (26%) of respondents find it consistently easy to browse• The advanced search page is used sparingly

Categories

• 62% of respondents would find categories consistently useful

Bookmarks

• 36% of respondents consistently return to previously found information• 71% of respondents use bookmarks “Most of the time” or “Always”• 65% of respondents consistently organize their bookmarks

6

OUCH!OUCH!

• “The search engine is poor to inadequate. I needed to find an appropriations data sheet and was returned 366 entries, none which had anything to do with appropriations. I spend far too much time looking through the search results for this engine to be effective. If I could find this document on the INTERNET I would do so, but this is an internal Raytheon document that is successfully hidden somewhere in the archives with the Ark of the Covenant.”

• Unidentified search and browse survey participant, June, 2003

• “Who gets more hits: www.amazon.com or www.thequaintbookstoredownthestreet.com? Listen up people: Our intranet is a wasteland of information. We need to unify - we need to standardize. Information is power - but only if it is on my desktop, not hidden away in some server waiting for a lucky adventurer to uncover it like some lost continent.”

• Another unidentified search and browse survey participant, June, 2003

7

Usability TestingUsability Testing

• Bentley College’s Design and Usability Testing Center

• 4 Focus Groups of 8-10 people each

• They told us:

• Want to filter searches

• Didn’t want long list of items to select from

• Liked “Suggested…” boxes

• Didn’t understand the taxonomy when presented like Yahoo!

• Liked taxonomy as file folder metaphor

• Liked thesaurus

• Were confused by relational taxonomies

• Liked “Categories” as the tab label, over Topics, Taxonomy, Thesaurus, Subjects or Browse.

8

Taxonomies – Who?Taxonomies – Who?

The Dream Team

Information Scientists

Cognitive Scientists

Linguists

Programmers

Database Experts

Network Specialists

Verity Administrators

Human Computer Interaction / Usability Experts

Subject Matter Experts

Organizational Change Management

What we got Information Scientists (in-house) Cognitive Scientist / Linguist (from Verity) Programmers, Database Experts, Network Specialists, and Verity

Administrators (both in-house and from Verity) HCI / UI Experts (in-house, Verity and Bentley College) Subject Matter Experts (in-house)

9

Taxonomies – What for?

• Browse & Navigation

• Relational Taxonomies

• Refine Search

• Parametric Search

• Federated Search

• Dynamic taxonomies

• Profiling

• Compliance Engine

• Classification & Categorization

• Provide controlled vocabularies to use with Metadata Schema(s)

• Easy selection to minimize angst over having to fill out file properties

10

Taxonomies 2003Taxonomies 2003

Deployed 5 taxonomies Defense Technologies (based on DTIC)

Purchased DTIC taxonomy Revised to fit Raytheon’s data Removed several categories including agriculture that are not needed

Raytheon Products Revised our products listing into a hierarchical approach Enlisted Raytheon Communicators as Subject Matter Experts

IPDS Built using data from the IPDS web site Enlisted IPDS experts as Subject Matter Experts

Engineering Implemented taxonomy built by Raytheon’s Engineering Technology

Network – Needs revising and enhancing Information Technology

Purchased from Verity Revised to fit Raytheon’s data Enlisted members of Corporate IT as Subject Matter Experts

11

Taxonomies 2004Taxonomies 2004

Human Resources taxonomy Deployed March 2004 Purchased from Verity Cross-functional team of HR representatives reviewed as Subject Matter

Experts Six Sigma

Will be deployed June 2004 Built based on Raytheon Six Sigma data

Legal Taxonomy Will be deployed June 2004 Purchased from Verity Able to create additional taxonomies for Ethics, Environmental Health &

Safety, and Export/Import Compliance from the purchase of this ONE taxonomy

Will be restructuring our top level categories : Business Units – domestic & international Functions Processes Products Relationships

12

Taxonomies – How?Taxonomies – How?

• Card sorting• EZ Sort• 3x5 cards

• Review search engine logs• Internal logs• Webtrends

• Review organic systems• Web and file share navigation

• Review existing taxonomies/thesauri• Concept Mapping

• Linguistic algorithm• Intelligent Classifier – lots of query building behind each node• MultiTes• Mind Manager• TextPad• BUY!

13

Build vs. BuyBuild vs. Buy

• Build to suit users• Reflect corporate vernacular• Internal acronyms• Corporate culture

• How is the business structured and portrayed?• Can take a long time

• Time estimates depend on type/use of taxonomy/tools available• Simple = 5 minutes to build term, 5 minutes to build category and

map the topic to the taxonomy• Complex = 75 minutes to build term• PLUS Quality Assurance testing!

• Buy• Industry standard• Rapid implementation

• Need customization• Both decisions require maintenance

14

Benefits - Increased Productivity

$0

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

$3,000,000

$3,500,000

$4,000,000

Dollars Returned to the Business for Growth(1 hour per year per general employee)

(1 hour per month per engineer)

Dollars $2,962,55$2,771,25$3,254,25$3,529,55$716,950$3,869,75$2,099,60$378,100$123,000

IDSIISRMSNCSRACSASRTSCRSLCorp

Delphi Group 2003 – as reported by Gartner• Business professionals spend more than 2 hours per day searching for information • Half of that time – 1 hour per day is wasted by failure to find what they seek• The single factor most attributed to the large amount of time wasted was

• data changes (location 35%) and • bad tools (ineffective search and lack of labeling 28%)

• If we conservatively assume only 1 hour per year would be saved per general employee and 1 hour per month per engineer, then:

0

1000

2000

3000

4000

5000

6000

7000

Hours Returned to Engineering by Business (1 hour per year per employee)

(1 hour per month per engineer)

Hours 44074319495653776616020287254269

IDSIISRMSNCSRACSASRTSCRSLCorp

15

Data growth assuming 60% annual growth rate

$10.8

$17.3

$27.7

$80.2

$19.6

$31.3

$50.1

$44.3

$12.2

$55.3

$33.2

2228340 544 870 1393$0

$10

$20

$30

$40

$50

$60

$70

$80

$90

2003 2004 2005 2006 2007

Mill

ion

s (A

nn

ual

Co

st)

0

500

1000

1500

2000

2500T1 OnlyGeneral tiered moveUnintelligent MovePolicy based Move

TB TB TB TB TB

Relative of starting point, growth curves represent storage acquisition cost increases over time.

Benefits – Reduce Storage CostsBenefits – Reduce Storage Costs

16

Is it working?Is it working?

• “New” search launched September 29

• Latest survey results show improvement

• Neutral rating upgraded to Good

• Metrics show increased usage of search

• 17% increase in unique users per day

• 25% increase in searches per day

• Metrics show increasing use of categories in search

• Since launch, the categories have been used 50,000 times

• ITLT approved project funding

• Knowledge Representation team recipients of 2003 IT Excellence in Collaboration and Knowledge Management Award

17

17

18

18

19

19

20

20

21

21

22

22