cloud hybrid search with sharepoint

60
Understanding and Applying Cloud Hybrid Search @jefffried Jeff Fried CTO, BA Insight

Upload: jeff-fried

Post on 22-Jan-2018

216 views

Category:

Internet


1 download

TRANSCRIPT

Page 1: Cloud Hybrid Search with SharePoint

Understanding and ApplyingCloud Hybrid Search

@jefffried

Jeff Fried CTO, BA Insight

Page 2: Cloud Hybrid Search with SharePoint

we love hybrid search - it's amazing how fast usage is growing

Jeff Teper @jeffteper

Page 3: Cloud Hybrid Search with SharePoint

Today’s Session

Page 4: Cloud Hybrid Search with SharePoint

Focused on Search and

SharePoint since 2004

Longtime

Search Nerd

• CTO, BA Insight

• Senior PM, Microsoft

• VP, FAST

• SVP, LingoMotors

About Jeff Fried

Passionate About

• Search

• SharePoint

• Search-driven

applications

• Information Strategy

Blog:

BAinsight.com/blog

Technet Column

“A View from the

Crawlspace”

[email protected]

Page 5: Cloud Hybrid Search with SharePoint

About BA Insight

– Connectivity

– Applications -

– Classification -

– Analytics

Page 6: Cloud Hybrid Search with SharePoint

6

Page 7: Cloud Hybrid Search with SharePoint
Page 8: Cloud Hybrid Search with SharePoint
Page 9: Cloud Hybrid Search with SharePoint

Demo

9

Page 10: Cloud Hybrid Search with SharePoint

The

Evolution

of

SharePoint:

HYBRID Management ExtensibilityExperiences

| Server

Experiences Management Extensibility

| Server | Server

HYBRID

Team

Sites

Portals

Enterprise

Content Mngt

BI

Page 11: Cloud Hybrid Search with SharePoint

Why Hybrid SharePoint?

Page 12: Cloud Hybrid Search with SharePoint

The Future of SharePoint Search, with Expert Jeff Fried

by Christian Buckley. March 23, 2015

Page 13: Cloud Hybrid Search with SharePoint

Today’s Session

Page 14: Cloud Hybrid Search with SharePoint

“Classic” Hybrid Search is Federated

not a single result set OOB

Page 15: Cloud Hybrid Search with SharePoint

Cloud Hybrid Search

Page 16: Cloud Hybrid Search with SharePoint

Access anywhere

Consistent user experience

Unified search results

No upgrades

No infrastructure mgt

Index storage scalable

Benefits of Cloud Hybrid Search

Page 17: Cloud Hybrid Search with SharePoint

Reduce Your Footprint

Servers

Volume of Content(indexable items) Pattern

On-prem Search Farm

Cloud Hybrid Search

0-10 million items Small 4 App + 2 DB 1 or 2

10-40 million items Medium 12 App + 2 DB 2

40-100 million items Large 28 App + 4 DB 2

400 million items XL example (SP2016) 86 App + 4DB 2 or 3

Page 18: Cloud Hybrid Search with SharePoint

SharePoint Server

(On-premises or Hosted)Office 365

SharePoint Online Content

Onedrive for Business ContentSharePoint Content

Cloud Hybrid Search

Page 19: Cloud Hybrid Search with SharePoint

SharePoint 2013/2016 Search Architecture

Web Service (CEWS)

Page 20: Cloud Hybrid Search with SharePoint

Walk-through: indexing & queries

SharePoint Server

(On-premises or Hosted)Office 365

Page 21: Cloud Hybrid Search with SharePoint

Today’s Session

Page 22: Cloud Hybrid Search with SharePoint

Case Study: Large University

Page 23: Cloud Hybrid Search with SharePoint

Setting up Cloud Hybrid Search

•Create

• Cloud Search Service Application in

SharePoint Server 2016

•Set up

• search architecture in SharePoint

Server 2016 for cloud hybrid search

•Connect

• your Cloud Search Service Application

to your Office 365 tenant

•Create

• a content source to crawl for cloud

hybrid search

•Setup

• Search Center to validate

hybrid search results in O365

•Start

• full crawl of on-premises

content for cloud hybrid search

•Verify• that cloud hybrid search works

Tune

• cloud hybrid search

experiences

Page 24: Cloud Hybrid Search with SharePoint

SupportSales & Marketing

Knowledge Articles

Fileshares

OneDrive

Support forum

SPO

Search Farm

SP 2013 content SP 2010 content

On-premises

Office 365

SPO content

SP 2013/2016

Cloud SSA

Example: Support Content

Page 25: Cloud Hybrid Search with SharePoint

Setup for Support Search

The Support Search vertical only searches sites that are relevant to the Support team.

It uses Local SharePoint results plus a filter on which sites to include in the search results

Result source query:

{searchTerms} (

Path:»http://sp2010» OR

Path:»file://fileshare» OR

Path:»http://demohybrid.../../supportforum»)

SharePoint Online Support Search

Page 26: Cloud Hybrid Search with SharePoint

Demo

26

Page 27: Cloud Hybrid Search with SharePoint

Search

Unified search across SharePoint on-premises and Office 365 content and people

SharePoint 2013/2016

Deliver unified search results

from Office 365 and on-

premises in a single search

Page 28: Cloud Hybrid Search with SharePoint

Search & discovery architecture wireframe --Online, on-premises, and hybrid

Page 29: Cloud Hybrid Search with SharePoint

External Content

(on-premises and/or

in the cloud)

SharePoint Server

(On-premises or Hosted)Office 365

SharePoint Online Content

Onedrive for Business Content

Co

nnect

ors

SharePoint Content

Adding External Content

Cloud Hybrid Search

Also drives:

• Office Graph (delve,..)

• Compliance (DLP, …)

Page 30: Cloud Hybrid Search with SharePoint

Connectors to Many Enterprise Systems• Aderant

• Amazon S3

• Alfresco

• Box

• Confluence

• CuadraSTAR

• Elite / 3E

• EMC Documentum

• EMC eRoom

• Google Drive

• HP Consolidated Archive

• (EAS, aka Zantaz)

• HPE Records Manager/HP TRIM

• IBM Connections

• IBM Content Manager

• IBM DB2

• IBM FileNet P8

• IBM Lotus Notes

• IBM WebSphere

• iManage Work

• Jive

• LegalKEY

• LexisNexis Interaction

• Lotus Notes Databases

• Microsoft Dynamics CRM

• Microsoft Exchange

• Microsoft Exchange Public Folders

• Microsoft SQL Server

• MySQL

• NetDocuments

• Neudesic The Firm Directory

• Objective

• OpenText LiveLink/RM

• OpenText eDOCS DM

• Oracle Database

• Oracle WebCenter

• Oracle WebCenter Content (UCM/Stellent)

• PLC/Practical Law

• ProLaw

• Salesforce.com

• SAP ERP

• ServiceNow

• SharePoint Online

• SharePoint 2016

• SharePoint 2013

• SharePoint 2010

• SharePoint 2007

• Sitecore

• Any SQL-based CRM system

• Veeva Vault

• Veritas Enterprise Vault

(Symantec eVault)

• West km

• Xerox DocuShare

• Yammer

Plus a proven architecture and process for creating new connectors to complex systems

Page 31: Cloud Hybrid Search with SharePoint

External Content in O365 UX

Unified view across all content

- on-premises and on-line

- inside and outside SharePoint

Page 32: Cloud Hybrid Search with SharePoint

Current Caveats:

1) don’t see thumbnails, just file icons

2) Have to query for it to show up

Page 33: Cloud Hybrid Search with SharePoint

External blog

SP OnPrem Yammer

Yammer

OneDrive SP Online

OneDrive

Page 34: Cloud Hybrid Search with SharePoint

Case Study: Cloud SSA, external content

Large global company

in materials science

Page 35: Cloud Hybrid Search with SharePoint

Today’s Session

Page 36: Cloud Hybrid Search with SharePoint

Issues with Cloud Hybrid Search (1)Cloud Hybrid Search "annoyances"

Performance Characteristicsslower query latency for on-prem queries against Cloud SSA

SharePoint Online Limitationsno synonyms

no site-level schema

no full trust code access

Hybrid Administration Weaknessesclunky metadata mapping

can't remove on-premises search results from Cloud SSA

trickier to test & debug crawls

can't reset index from Cloud SSA

Be aware of these

& compensate for them

(Fixed in August PU)

(Semi-addressed in June PU)

And it’s getting better:

Page 37: Cloud Hybrid Search with SharePoint

2017

Page 38: Cloud Hybrid Search with SharePoint

38

Performance

Page 39: Cloud Hybrid Search with SharePoint
Page 40: Cloud Hybrid Search with SharePoint

https://<<tenant_name>>-admin.sharepoint.com/_layouts/15/searchadmin/TA_SearchAdministration.aspx

Page 41: Cloud Hybrid Search with SharePoint

Item Limits and Pricing

Licensing: 1M items of external content in index for every 1TB storage in O365

1TB included by default

+ 0.5 GB per licensed O365 user

No limit on number of items from O365 in the index

Default throttling at 20M external items; current threshold at 25M

2000 users x 0.5 GB = 1TB

+ 1TB default = 2 TB total

-> 2M external items indexed

+ Can also buy the “Office 365 Extra File Storage” Add-on

$0.20/GB/Month = $200/TB/Month = $200/M items/Month

50,000 users x 0.5 GB = 25TB

+ 1TB default = 26 TB total

-> 26M external items indexed

Page 42: Cloud Hybrid Search with SharePoint

Should I run index reset?

NO!DeleteAllCloudHybridSearchContent()

https://blogs.technet.microsoft.com/beyondsharepoint/2016/07/07/cloud-hybrid-search-service-application-removing-items-from-the-office-365-search-index/

Page 43: Cloud Hybrid Search with SharePoint

Issues with Cloud Hybrid Search (2)

43

Content Enrichmentno CEWS

no Entity Extraction

Securityno Custom Security Trimming

Can't crawl across Multiple Domains

Can't Crawl SP in Classic Auth Mode

Data Sovereigntyexport-restricted content

can't be put in O365 index

Limitations of Cloud SSA

Page 44: Cloud Hybrid Search with SharePoint

External Content

(on-premises and/or

in the cloud)

SharePoint Server

(On-premises or Hosted)

SPO Content

OneDrive Content

Co

nnect

ors SharePoint Content

Connector

Framework

Office 365

AutoClassifier

(app version)

CEWS

Custom

Processing

Page 45: Cloud Hybrid Search with SharePoint

Case study:Content Enrichment

Content

CloudSSA

Connector Framework

IndexingConnectors

Smart Pipeline

AutoClassifierCustom Stage A

CustomStage C

Custom Stage B

Page 46: Cloud Hybrid Search with SharePoint

Online

On-Prem

Cloud Hybrid Search under the coversSecurity = identity sync + ACL mapping

Cloud SSACloud SSA

ParseCrawl

SCS

ACL Map Process

Blob

storequeue

Page 47: Cloud Hybrid Search with SharePoint

Directory Synchronization

SID S-1-5-21-1212121212-1212121212-1212

[email protected]

msOnline-

OnPremiseSecurity

Identifier

S-1-5-21-1212121212-1212121212-1212

PUID PUID-XXXX-XXXXXXXXXX

Page 48: Cloud Hybrid Search with SharePoint

Mapping of Access Control Lists

Allow: S-1-5-21-1212121212-1212121212-1212 Allow: PUID-XXXX-XXXXXXXXXX

• User SIDs are mapped to PUIDs

• Group SIDs are mapped to Object IDs

• «Everyone» and «Authenticated users» are mapped to

«Everyone except external users»

Only AD Users and Groups,

Only from one domain

Page 49: Cloud Hybrid Search with SharePoint

Case Study: Crawling Cross-Domain

Page 50: Cloud Hybrid Search with SharePoint

A global single index solution

Cloud SSA

Cloud SSA

Cloud SSA

Cloud SSA

Cloud SSA

BUT export-restricted content

can’t be in the global index

Page 51: Cloud Hybrid Search with SharePoint

Hybrid searchFederated search

Azure

Page 52: Cloud Hybrid Search with SharePoint

Issues with Cloud Hybrid Search OOB

Content Enrichmentno CEWS

no Entity Extraction

Securityno Custom Security Trimming

Can't crawl across Multiple Domains

Can't Crawl SP in Classic Auth Mode

Data Sovereigntyexport-restricted content

can't be put in O365 index

Limitations of Cloud SSA BA Insight Solution

Connector Framework

AutoClassifier

Connector Framework

can 'map down' to AD groups

can 'map across' cross-domain

can crawl and map security

Federator

Page 53: Cloud Hybrid Search with SharePoint

Today’s Session

Page 54: Cloud Hybrid Search with SharePoint
Page 55: Cloud Hybrid Search with SharePoint

Federated / Hybrid

Compliance

Constraints

Desired UX

Page 56: Cloud Hybrid Search with SharePoint

Complexity of environment

A/A, Trusts and Federation

Extension of SP farm design

Skill-set required• Identity, Security

• Networking

• SP Infrastructure

• Information mgmt. design

Page 57: Cloud Hybrid Search with SharePoint
Page 58: Cloud Hybrid Search with SharePoint
Page 59: Cloud Hybrid Search with SharePoint