search solutions can be broad and...

Search solutions can be broad and deep

Productivity (Intranet) Search Experience

Search Driven Applications

Connect all of your people with a broad set of information

• Enterprise content and data • Intranet sites • Team workspaces • People and expertise

Drive measurable ROI by helping a business group make the most

of a specific set of information

• 360o customer insight

• Competitive intelligence • Research portals

Ingredients for great search

Search Driven Applications put all of the pieces together

Search connects people to the information

they need to get their jobs done.

Search Driven Applications drive measurable ROI by helping a specific set of people make the most of a specific set of information.

Forrester View of Search-Based Apps

Desired characteristics: • Highly scalable

• Quick and agile development

• Adaptable to diverse, dynamic

information sources (web, enterprise,

data, content, email, etc.)

• Intuitive for targeted audience

• Heterogeneous information is

harmonized

Query

Semantic

search

Query builders

Explore and

combine

Faceted

interaction

Non-modeled

Topics and

dimensions

Visualize

Reporting

Dashboards

Relationship

Map

Act

Advanced

analytics

Scorecards

Process triggers

Search-Driven Applications Meet all the search application needs you have across your business

Sales:

360o Customer Insight

Services:

Knowledge Browser

Marketing:

Competitive Intelligence

Research & Development:

Innovation Portal

Support:

Call Center Advisor

Operations:

Systems/Logistics Portal

“How do I support the

unique search needs of teams and

work that impact our business?”

To do so, you need a search platform that has

A deep understanding

of your information

Flexible relevance to meet

diverse needs

A customizable UX to increase

user efficiency

Customized productivity search

and search driven applications

Social, Conversational++,

Visual, Contextual

Connectivity+, Security,

Relevance++, Performance,

Administration+, Scale++

General Productivity search

Light customization

Social, Conversational

Connectivity, Security,

Relevance, Performance,

Administration, Scale

SharePoint 2010 search and FAST Search for SharePoint - Comparison

Customizations for Search Driven Applications

Building on an extensible platform

Configure Extend Create

User Context

LOB Connectivity

Content Processing

Business language

Federation Sources

UI Look & Feel

…..

Relevance Profiles

UI & Web Parts

Result Rollup

Visual Elements

Workflows

Analytics

…..

Custom Elements

Work Environments

New Innovations

….

Out of the box features

SharePoint & FAST Custom federation runtimes

Custom refinement filter generators

Query or results alteration

Connector Framework

Custom applications leveraging Web Service / RSS

FAST-specific features Contextual Search

Document Preview (results with thumbnails)

Visual Best Bets

Sort on any property

Broader query syntax (FQL)

Similar Search

Enhanced language support

Knowledge Center for Services: Contoso Consulting

Meet Arlene Huff Senior Strategy Consultant

Delivers deep, insightful advice to clients

Depends on knowledge and experience across Contoso

Arlene gets a request for a meeting with a new prospect

Blue Yonder Airlines wants help in developing a strategy for

improving their logistics and supply chain and wonders how

to fit collaboration and knowledge management into this.

They want a meeting tomorrow – 9AM!

Arlene needs to pull together a presentation and a set of experts for the meeting - FAST.

Most information is in PowerPoint and in consultant’s heads

Logistics, supply chain, collaboration & knowledgement

management, and strategy are all separate areas at Contoso

Across thousands of engagements and hundreds of experts,

what and who match Blue Yonders’ situation and request

most closely?

Knowledge Center

Knowledge Center

collaboration KM strategy


Knowledge Center You searched for: collaboration KM strategy > 1996 to 2007 Travel Key Concepts Service Offerings



Recognize useful content within a consulting


Knowledge Center You searched for: collaboration KM strategy > 1996 to 2007 Travel Key Concepts Service Offerings Which repositories do

I want to explore?

Which file formats?

When and where have we done work like this?



Select elements to explore and narrow in



Is there important news I can share with the client?



Which experts are available for a meeting?



Which experts are available for a meeting?

Sure, send me a draft ppt, ok?



Build a presentation as I explore



Start a workflow immediately

How would you create this?

Content Crawling: bring in data from lots of places

OOB connectors to SharePoint (reports, account documents), shared

files; CMS systems across multiple divisions of contoso

Crawl web intelligently for background content

Content processing: creating metadata

Names of projects, offerings, key concepts, clients, external experts

Industry terms & taxonomy (from external sources)

Synonyms for key concepts

OOB web parts configured for style

Federation, People Search, Search actions, Scopes

Custom Relevance Profile

Custom web parts for visual navigation

Tab-like selectors, sliders, maps, and taxonomy-aware refiners

Build PowerPoint on the Fly Custom development using the OpenXML SDK

SharePoint workflows for act-on-selected-items

Search 2010 Architecture The platform for Search Customization

FAST Search for SharePoint Designed for Customization

… …

Metadata

Creation

Relevance

Control

User

Context

Indexing

Connectivity

User

Experience

Federation

Extending Search Web Parts


All Web Parts communicate through common interface Possible to add new Web Part on the page that interacts

with existing Web Parts. i.e. Graphical refiner, Tag cloud Refiner, etc…




FAST Search Web Parts extend SharePoint Foundation Search Web Parts




FAST Search Web Parts extend SharePoint Foundation Search Web Parts

Core Search Web Parts are unsealed

Web Part Options

Web Part Options

Configure Modify Web Part Properties

Modify Site and Farm Settings

Master Pages, Site Pages, CSS, Theming

Web Part Options




Extend Custom Actions

Visualizations

Web Part Options




Extend Custom Actions

Visualizations

Create Example: Multi-select refinements

Other Custom Web Part Examples


Actions Get more results

Save Search

Export Search


Actions Get more results

Save Search

Export Search

Graphical Refiners Pie/Bar/Column charts

Range Refiners

Tunable Relevance Enables unique, business-specific search results for diverse roles

Rank Profiles Tune relevancy without impacting the default algorithm

Quality Also known as static rank, consists of multiple managed

properties including site, URL depth (preference for shorter

URLs), and relative importance of links to this document.

Authority Applies when the query word falls in the link or anchor text.

Query

Authority

Maps the popularity of a document, or the click-through rate

when documents are clicked as a result of a query

Freshness Increases the relevancy if a document was recently created

or modified, based on the last modified property.

Proximity Applies to where query terms fall and how close they are to

each other within a document

Context Increases the rank of a document if the query term is a

managed property associated with that document

Managed

Property

Effects relevancy when a managed property contains a

specific value, such as Woodgrove Bank or Financial Services

Out of the box relevancy Tuned for great general productivity experience,

relevancy improves with click-throughs and link text

analysis.

Extend the default algorithms Create new default relevancy models. Blend static and

dynamic ranking parameters to instantly improve

search results.

# In FAST Search Server for SharePoint 2010 shell # Retrieve the default rank profile $rp = Get-FASTSearchMetadataRankProfile -Name default # Create a new rank profile called freshness $np = New-FASTSearchMetadataRankProfile -Name freshness -Template $rp # Increase the freshness.weight to 10000 $np.FreshnessWeight = 10000 # Update your profile $np.Update()

How to create a Rank Profile

Rank Profiles created

in PowerShell by

extending the default

relevancy algorithm…

… and are exposed in the

user interface by modifying

the sorting web part

IT Pros are empowered to create new profiles quickly

1.

2.

Query Language Options

SharePoint Server 2010

FAST Search for SharePoint –

FQL SharePoint Server Search

Keyword Syntax 1

2

AND /+, OR, NOT/-

a NEAR b

ALL(…), ANY(…), NONE(…)

title:xxx, scope:person

Keyword Query Language (KQL)

FQL provides a robust and expressive query language

Wildcard support - *, ? Numeric Data types (Integer, Float, Decimal, Datetime)

Operators Direct field access (e.g., title:othello, author:shakespeare) Numeric (COUNT, RANGE, <, <=, >, >=) Boolean (AND, OR, ANY, NOT) Rank (RANK, XRANK) Proximity (NEAR, ONEAR) Sorting (SORT, SORTFORMULA)

String (operator support for strings) Boundary (starts-with, ends-with, equals) Filter

FAST Query Language

[property-spec:]:operator(operand [,operand]* [,

parameter="value"]*)

FQL Operators

Query Operator: A keyword that specifies an operation to perform

Operator Type Keywords

Boolean AND, ANDNOT, OR, ANY, NOT, COUNT, RANK, XRANK

Proximity NEAR, ONEAR

Numeric FLOAT, INT, DATETIME

String WEIGHT, WILDCARD, MODE

Boundary Match STARTS-WITH, ENDS-WITH, EQUAL

Query Language Expressiveness Soft boost using the Xrank operator

Boost all documents that contain the name

Or, in the FAST Query Language:

Single (documents containing either Mary, Diane or both gets a boost of 5000):

Xrank(string(”performance”), or(person:string(”diane tibbott”), person:string(”mary baker”)),

boost=5000))

Accumulative (documents with both Mary and Diane gets a boost of 10000):

Xrank(Xrank(string(”performance”), person:string(”diane tibbott”), boost=5000),

person:string(”mary baker”), boost=5000)

Diane

Tibbott

Boost all documents that contain the name

Mary

Baker

Federation

Search

Provider

Search

Client

{searchTerms}

HTTP request

Search

Provider

Search

Client

{searchTerms}

<title> <link> <description>

HTTP request

RSS/Atom

Results

Search

Provider

Search

Client

{searchTerms}


<pubdate> <author> <category> <media:thumbnail>

HTTP request

RSS/Atom

Results

Search

Provider

Search

Client

{searchTerms}


<pubdate> <author> <category> <media:thumbnail>

<recordid> <projectname> <contactnumber>

HTTP request

RSS/Atom

Results

Search

Provider

Search

Client

existing searchable feed or create your own

OpenSearch

Page

Search

Provider

Search

Client

FAST Search for SharePoint

Document Processor Pipeline (Language detection, document conversion,

tokenization, entity extraction, schema

mapping)

Document Processing - Pipeline

Stage

1

Stage

2

Stage

...

Stage

N

Search

Index

Connectors

(SharePoint,

File Shares,

Web...)

Documents

A pipeline is a sequential set of processing stages

Their job is to break down and enrich content for indexing

They perform functions such as applying linguistics to optimize content for search

Or parsing content to extract additional metadata

Custom stages can be also created and added to the pipeline by a developer

Document Processing - Pipeline Enhance your content for optimal search experience and findability







Format

Conversion

Extracts plain text and metadata from multiple content

formats (e.g. Microsoft Office, PDF, HTML, etc.)







Language

Encoding and

Detection

Identifies the encoding and languages used in the text

content so that the appropriate linguistic normalization

rules and dictionaries can be applied downstream







Tokenization

Breaks text into tokens using language-specific rules for

punctuation, diacritics, accents, compound words, phrases

and numbers (currency, telephones, part numbers, etc.)







Lemmatization

Applies language-specific normalization to content so

users’ queries match words and phrases in canonical or

inflected forms (singular/plural, masculine/feminine, etc.)







Property

Extraction

Recognizes predefined entities mentioned in the content;

out of the box support for Companies, Locations and

People but this can be extended to other categories







Vectorization

Creates document vectors (phrase/weight pairs reflecting

important terms and frequency of occurrence) to enable

“find similar” functionality







Date and Time

Normalization

Converts dates and times to a standard representation, to

handle locale-specific representations; for example, the

date 14-Mar-10 is equivalent to March 14, 2010







Custom

Processing

Stage

Enables you to extend the content processing pipeline

with custom stages (home-grown solutions or 3rd party

software) to address your own business needs







Properties

Mapper

Maps the relevant pieces of content and metadata

discovered in the pipeline to the index schema for search

Powerful Entity Extraction Enables search-driven navigation that is relevant to your business

CONCEPT

PRODUCT

COMPANY

Custom Entities & Refiners

PRODUCT (Custom)

CONCEPT (Custom)

COMPANY (OOTB)

What do you need to do in the Backend? 1. Create string property 2. Create and deploy dictionary

Put your terms in the out of the box Extraction dictionaries by modifying an XML file

3. Activate extractor Map the crawled property to a managed property

4. (Re-)Feed / Index content

What do you need to do in the Front-end:

Modify refinement panel web part

1. Define product refiner 2. Enable product refiner

Create a New Refiner based on Property Extraction

Custom Entity Extraction

Custom entity extractors may be created

All based off of dictionary files you provide

Extractors can be ‘wholeword’ extractor

Or can be ‘wordpart’ (substring) extractors

Use ‘wordpart’ extractors for Chinese

Add Custom Processing

Content classification

Geo-tagging

Machine translation

Sentiment Analysis

Extending Pipeline capabilities Safely add additional analysis and processing

Add Custom Processing

Content classification

Geo-tagging

Machine translation

Sentiment Analysis

Extending Pipeline capabilities Safely add additional analysis and processing

What is Custom Processing? Pipeline Extensibility is a specially defined stage that

takes a set of crawled properties, as flat text as input

and maps output to another crawled property

Custom Processing is Safe Executable arguments and temporary files

automatically handled in sandbox with timeouts.

Runs before Crawled Property mapping stage,

making new metadata accessible in SharePoint

Pipeline Extensibility - Detail

Why an Extensible Pipeline?

Populate document meta data using custom code

Integrate with 3rd party software additional functionality - Examples:

Sentiment Analysis

Taxonomy classification

Entity extraction

Document translation

Video transcription

Thumbnail generation

Alerting

Custom statistics or monitoring of crawled documents

Pipeline Extensibility - Process

Determine input and output crawled properties

Create executable that takes in 2 arguments.

Arg 1 - In file name

Arg 2 – Out file name

Deploy to C:\FastSearch\bin

Modify pipelineextensibility.xml file

Specify executable

Specify input and output crawled properties

Create and map any crawled and managed properties

Restart Document Processors

Crawl Full

Pipeline Extensibility: Configuration

<PipelineExtensibility> <Run command="sample.exe %(input)s %(output)s"> <Input> <CrawledProperty propertySet="f29f85e0-4ff9-1068-ab91-08002b27b3d9" varType="31" propertyId="4"/> </Input> <Output> <CrawledProperty propertySet="d5cdd505-2e9c-101b-9397-08002b2cf9ae" varType="31" propertyName="MyProperty"/> </Output> </Run> </PipelineExtensibility>

PS> Get-FASTSearchMetadataCrawledProperty -name fileextension CategoryName : Basic IsMappedToContents : False IsNameEnum : False IsMultiValued : False Name : FileExtension Propset : 0b63e343-9ccc-11d0-bcdb-00805fccce04 VariantType : 31

<Document> <CrawledProperty propertySet="f29f85e0-4ff9-068- ab91-08002b27b3d9" varType="31" propertyId="4"> John Doe </CrawledProperty> </Document>

Input/Output File Format Get Property Attributes

Limitations

Back End Processing Tasks:

Load content from many different places Out of the box connectors for SharePoint, exchange public folders, and shared files

SharePoint Designer to configure connection to customer portfolio/holdings database

Create custom metadata with content processing pipeline

Names of holdings, offerings, key concepts, companies, people

Synonyms for key concepts (real estate ~ REIT)

Roll-ups configured with optional results collapsing stage

Create custom relevance profile

Designers can stylize the User Interface

Apply styles to web parts Federation, People Search, Search actions

Build custom web parts for visual navigation

Use SharePoint workflows to perform business specific actions

Leveraging FAST to build applications Putting together all of the pieces to build search-driven applications

Search Driven Application Advantages

IDC: “Their selling point is that a worker can sit down and

accomplish a job without having to move from one information

source to another, or from one application to the next. These

full-blown applications will develop integrated work

environments in which the UI design hides the complexity of

multiple information sources and applications. Our work with

users in the marketplace convinces us that applications that are

affordable, easy to deploy, streamline work, enlighten, and

reveal are well loved, and therefore solidly enthroned in an

organization.”

Source: Worldwide search and text analytics 2010 top 10 predictions 01/2010,

Worldwide search and discovery software 2009-2013 forecast update and 2008 vendor shares, 01/2009

http://mslibrary/research/MktResearch/IDC/Intranet/221799.htm




