fried connecting across silos seminar

Connecting across silos Using search to integrate O365, Dynamics 365,

SharePoint, and LOB systems

Agenda

2

Connecting across silos Using search to integrate O365, Dynamics 365,

SharePoint, and LOB systems

@jefffried

Jeff Fried CTO, BA Insight

4

Focused on Search and

SharePoint since 2004

Longtime

Search Nerd

• CTO, BA Insight

• Senior PM, Microsoft

• VP, FAST

• SVP, LingoMotors

About Jeff Fried

Passionate About

• Search

• SharePoint

• Search-driven

applications

• Information Strategy

Blog:

BAinsight.com/blog

Technet Column

“A View from the

Crawlspace”

jeff.fried@bainsight.com

About BA Insight

– Connectivity

– Applications -

– Classification -

– Analytics

We live in Silos

of knowledge workers regularly access

4 or more systems to get the

information they need to do their jobs

61%

regularly access 11 or more systems

of a typical knowledge worker’s day is spent

looking for and consolidation information

spread across a variety of systems

36%

15%

The average $1 billion company maintains 48 disparate financial systems and uses 2.7 ERP systems

Integration Gaps Impact Performance

Source: The Hackett Group

Example 1: (Pharma R&D) - silos

1) Research was done in separate group

• Seemingly unrelated research projects

• Later in lifecycle (mfg, reg/test)

2) Data not accessible

• Isolated content source

• Restricted / limited access

• Source not searchable

• Special knowledge required

3) Data not linked

• Various names/changes leave data disconnected

• People not connected to data (experts)

• Data managed in many unconnected systems

The problem is getting worse

–

–

SalesForce twitterYou

TubeLinkedIn RSS

Google

Docs

BaseCampDropBoxWeb Hosting

Yammer

SharePointExchange

File Servers

Alfresco

CRMERP

Planning

Portfolios

Unless users knew where to look for content, they were not finding relevant information

Besides inefficiency and user frustration, this resulted in a proliferation of duplicates, and big security holes

-> a vicious cycle

Example 2: (non-profit) – proliferation of systems

Even within integrated suites, there is lots of dis-integration

Silobuster approaches

–

–

–

–

Search Provides a Unified View

–

–

–

Using Search for Information Integration

Demo

21

What is an indexing connector?

Enterprise

Repositories

Connector

Connect Securely1

Map Content2

Feed Engine3

Find any changes4

Search

Engine

Content

ContentProcessing

SearchIndex

SearchUI

QueryProcessing

Connectors

Context

LOB Systems

Pattern: Single Search Index

Example 2: (Pharma R&D) Unified View

1. Documentum Image

2. SharePoint Doc

3. Regulatory Record

4. MEDLINE article

Multiple Sources One Search

Search: amgen 655

Relationships Discovered:

Antibodies: mAb

Receptors: DR5, IGF-1R

Labs: Oncology 1

People: David Chang

Example 2 (on-profit) – iterative solution

Consolidate

into 2 Cloud + 3 on-premises repositories

Connect to create a unified view

Standardize

with metadata

–

–

–

BA Insight Approach

Connectivity

Classification

Applications

Analytics

Search

Engines

Environments

BA Insight Software Portfolio

27

Connectivity – Secure Connectivity to over 60 Enterprise Systems

Classification - Auto-Classify Content Accurately

Applications – Boost Productivity by Finding Information Faster

Analytics – Use Analytics to improve the User Experience

Environments - Compelling user experiences running in

SharePoint, O365, Hybrid SharePoint & Salesforce.

Search Engines - Support for multiple search engines, including

SharePoint, Elasticsearch, Decisiv, Office 365 & cloud hybrid search

28

Connectors to Many Enterprise Systems• Aderant

• Amazon S3

• Alfresco

• Box

• Confluence

• CuadraSTAR

• Elite / 3E

• EMC Documentum

• EMC eRoom

• Google Drive

• HP Consolidated Archive

• (EAS, aka Zantaz)

• HPE Records Manager/HP TRIM

• IBM Connections

• IBM Content Manager

• IBM DB2

• IBM FileNet P8

• IBM Lotus Notes

• IBM WebSphere

• iManage Work

• Jive

• LegalKEY

• LexisNexis Interaction

• Lotus Notes Databases

• Microsoft Dynamics CRM

• Microsoft Exchange

• Microsoft Exchange Public Folders

• Microsoft SQL Server

• MySQL

• NetDocuments

• Neudesic The Firm Directory

• Objective

• OpenText LiveLink/RM

• OpenText eDOCS DM

• Oracle Database

• Oracle WebCenter

• Oracle WebCenter Content (UCM/Stellent)

• PLC/Practical Law

• ProLaw

• Salesforce.com

• SAP ERP

• ServiceNow

• SharePoint Online

• SharePoint 2016

• SharePoint 2013

• SharePoint 2010

• SharePoint 2007

• Sitecore

• Any SQL-based CRM system

• Veeva Vault

• Veritas Enterprise Vault

(Symantec eVault)

• West km

• Xerox DocuShare

• Yammer

Seamless Connectivity

Single point of access

–

–

–

–

Secure Connectivity to Enterprise Systems

Provide unified views of all knowledge assets

–

–

–

–

–

–

Metadata Drives Great User Experiences

Documents from many sourcesAll client or matter-relevant documents are integrated.

Rich MetaDataContent annotated automatically – concepts,

categories, citations, matters, clients, etc

Navigation ControlsExplore, Discover, Drill-down

Manual Tagging is impractical

and remarkably inconsistent

Automation

Called: AutoClassification, AutoTagging, Metadata Generation, Text Analytics, ….

Demo

34

The

Evolution

of

SharePoint:

HYBRID Management ExtensibilityExperiences

| Server

Experiences Management Extensibility

| Server | Server

HYBRID

Team

Sites

Portals

Enterprise

Content Mngt

BI

Cloud Hybrid Search

Demo

42

SharePoint 2013/2016 Search Architecture

Web Service (CEWS)

Reduce your footprint

Servers

Volume of Content(indexable items) Pattern

On-prem Search Farm

Cloud Hybrid Search

0-10 million items small 4 App + 2 DB 1 or 2

10-40 million items medium 12 App + 2 DB 2

40-100 million items large 28 App + 4 DB 2

400 million items XL example (SP2016) 86 App + 4DB 2 or 3

SharePoint Server

(On-premises or Hosted)Office 365

SharePoint Online Content

Onedrive for Business ContentSharePoint Content

Cloud Hybrid Search

Case Study: Split Users with SharePoint

SupportSales & Marketing

Knowledge Articles

Fileshares

OneDrive

Support forum

SPO

Search Farm

SP 2013 content SP 2010 content

On-premises

Office 365

SPO content

SP 2013/2016

Cloud SSA

External Content

(on-premises and/or

in the cloud)

SharePoint Server

(On-premises or Hosted)Office 365

SharePoint Online Content

Onedrive for Business Content

Co

nnect

ors

SharePoint Content

Adding External Content

Cloud Hybrid Search

Also drives:

• Office Graph (delve,..)

• Compliance (DLP, …)

External Content in O365 UX

Unified view across all content

- on-premises and on-line

- inside and outside SharePoint

Current Caveats:

1) don’t see thumbnails, just file icons

2) Have to query for it to show up

–

–

–

–

Case Study: Cloud SSA, external content

Large global company

in materials science

Issues with Cloud Hybrid Search (1)Cloud Hybrid Search "annoyances"

Performance Characteristicsslower query latency for on-prem queries against Cloud SSA

SharePoint Online Limitationsno synonyms

no site-level schema

no full trust code access

Hybrid Administration Weaknessesclunky metadata mapping

can't remove on-premises search results from Cloud SSA

trickier to test & debug crawls

can't reset index from Cloud SSA

Be aware of these

& compensate for them

(Fixed in August PU)

(Semi-addressed in June PU)

And it’s getting better:

Issues with Cloud Hybrid Search (2)

54

Content Enrichmentno CEWS

no Entity Extraction

Securityno Custom Security Trimming

Can't crawl across Multiple Domains

Can't Crawl SP in Classic Auth Mode

Data Sovereigntyexport-restricted content

can't be put in O365 index

Limitations of Cloud SSA

External Content

(on-premises and/or

in the cloud)

SharePoint Server

(On-premises or Hosted)

SPO Content

OneDrive Content

Co

nnect

ors SharePoint Content

Connector

Framework

Office 365

AutoClassifier

(app version)

CEWS

Custom

Processing

Case study:Content Enrichment

Content

CloudSSA

Connector Framework

IndexingConnectors

Smart Pipeline

AutoClassifierCustom Stage A

CustomStage C

Custom Stage B

Case Study: Crawling Cross-Domain

A global single index solution

Cloud SSA

Cloud SSA

Cloud SSA

Cloud SSA

Cloud SSA

BUT export-restricted content

can’t be in the global index

Issues with Cloud Hybrid Search OOB

Content Enrichmentno CEWS

no Entity Extraction

Securityno Custom Security Trimming

Can't crawl across Multiple Domains

Can't Crawl SP in Classic Auth Mode

Data Sovereigntyexport-restricted content

can't be put in O365 index

Limitations of Cloud SSA BA Insight Solution

Connector Framework

AutoClassifier

Connector Framework

can 'map down' to AD groups

can 'map across' cross-domain

can crawl and map security

Federator

A new wave of Intelligent Search / Intranets?

Relevant: Deliver all content users need, independent of where it resides

Proactive: Intelligently and automatically deliver content in context

Interactive: Present users with familiar search patterns

Pillars of intelligent search

All of these:

• Depend on being Connected, Structured, Personalized, & Adaptive

• Use NLP and ML in addition to traditional search technologies

–

–

–

–

–

Turning Microsoft Search into an Insight Engine

Demo

66

Office 365USER

EXPERIENCES

Unified Search IndexSEARCH INDEX

Office 365DATA

REPOSITORIES

Search & discovery architecture --Online, on-premises, and hybrid

Office 365MICROSOFT

GRAPH

Cloud-hosted, on-premises &

mobile apps

On-premises

Content

ContentProcessing

SearchIndex

Search UI

ML with BA Insight

QueryProcessing

AutoClassifierFlexible combination of Rules & ML

FederatorPluggable query and results pipelines

Analytics

Smart AnalyticsBehavioral data for insight and learning

–

What is Natural Language Processing?

Complicators

–

–

–

–

–

Content

ContentProcessing

SearchIndex

Search UI

NLP with BA Insight

QueryProcessing

AutoClassifierFlexible combination of Rules & ML

FederatorPluggable query and results pipelines

Analytics

Smart AnalyticsBehavioral data for insight and learning

AutoClassifierrules-driven core, combined with learning-based modules

Enhanced Content

Enriched with

Metadata and

Content Types

Search Visualization Workflow

Federator: pluggable query & results processing

•

•

•

•

Content

ContentProcessing

SearchIndex

Search UI

ML applied to search

QueryProcessing

Learning about content structure Learning about user intent

Analytics

Behavioral data to drive learning

Learning to rank

79

Search practices are improving

Why Can’t Everything in an Intranet be Dynamic?

• News

• Forums

• Content

• Navigation

• Tools/Applets

• Analytics IT CAN

Nearly every Intranet will need to add

1. ConnectivityConnectors to content & systems outside of O365

3. ContextPersonalization through behavioural information

2. Structure Information Architecture & Tags for navigation &

findability

Smart Content needs Gardeners

Contact:Jeff.Fried@BAinsight.comwww.BAinsight.com

Questions