fried connecting across silos seminar
TRANSCRIPT
Connecting across silos Using search to integrate O365, Dynamics 365,
SharePoint, and LOB systems
Agenda
2
Connecting across silos Using search to integrate O365, Dynamics 365,
SharePoint, and LOB systems
@jefffried
Jeff Fried CTO, BA Insight
4
Focused on Search and
SharePoint since 2004
Longtime
Search Nerd
• CTO, BA Insight
• Senior PM, Microsoft
• VP, FAST
• SVP, LingoMotors
About Jeff Fried
Passionate About
• Search
• SharePoint
• Search-driven
applications
• Information Strategy
Blog:
BAinsight.com/blog
Technet Column
“A View from the
Crawlspace”
About BA Insight
– Connectivity
– Applications -
– Classification -
– Analytics
We live in Silos
of knowledge workers regularly access
4 or more systems to get the
information they need to do their jobs
61%
regularly access 11 or more systems
of a typical knowledge worker’s day is spent
looking for and consolidation information
spread across a variety of systems
36%
15%
The average $1 billion company maintains 48 disparate financial systems and uses 2.7 ERP systems
Integration Gaps Impact Performance
Source: The Hackett Group
Example 1: (Pharma R&D) - silos
1) Research was done in separate group
• Seemingly unrelated research projects
• Later in lifecycle (mfg, reg/test)
2) Data not accessible
• Isolated content source
• Restricted / limited access
• Source not searchable
• Special knowledge required
3) Data not linked
• Various names/changes leave data disconnected
• People not connected to data (experts)
• Data managed in many unconnected systems
The problem is getting worse
–
–
SalesForce twitterYou
TubeLinkedIn RSS
Docs
BaseCampDropBoxWeb Hosting
Yammer
SharePointExchange
File Servers
Alfresco
CRMERP
Planning
Portfolios
Unless users knew where to look for content, they were not finding relevant information
Besides inefficiency and user frustration, this resulted in a proliferation of duplicates, and big security holes
-> a vicious cycle
Example 2: (non-profit) – proliferation of systems
Even within integrated suites, there is lots of dis-integration
Silobuster approaches
–
–
–
–
Search Provides a Unified View
–
–
–
Using Search for Information Integration
Demo
21
What is an indexing connector?
Enterprise
Repositories
Connector
Connect Securely1
Map Content2
Feed Engine3
Find any changes4
Search
Engine
Content
ContentProcessing
SearchIndex
SearchUI
QueryProcessing
Connectors
Context
LOB Systems
Pattern: Single Search Index
Example 2: (Pharma R&D) Unified View
1. Documentum Image
2. SharePoint Doc
3. Regulatory Record
4. MEDLINE article
Multiple Sources One Search
Search: amgen 655
Relationships Discovered:
Antibodies: mAb
Receptors: DR5, IGF-1R
Labs: Oncology 1
People: David Chang
Example 2 (on-profit) – iterative solution
Consolidate
into 2 Cloud + 3 on-premises repositories
Connect to create a unified view
Standardize
with metadata
–
–
–
BA Insight Approach
Connectivity
Classification
Applications
Analytics
Search
Engines
Environments
BA Insight Software Portfolio
27
Connectivity – Secure Connectivity to over 60 Enterprise Systems
Classification - Auto-Classify Content Accurately
Applications – Boost Productivity by Finding Information Faster
Analytics – Use Analytics to improve the User Experience
Environments - Compelling user experiences running in
SharePoint, O365, Hybrid SharePoint & Salesforce.
Search Engines - Support for multiple search engines, including
SharePoint, Elasticsearch, Decisiv, Office 365 & cloud hybrid search
28
Connectors to Many Enterprise Systems• Aderant
• Amazon S3
• Alfresco
• Box
• Confluence
• CuadraSTAR
• Elite / 3E
• EMC Documentum
• EMC eRoom
• Google Drive
• HP Consolidated Archive
• (EAS, aka Zantaz)
• HPE Records Manager/HP TRIM
• IBM Connections
• IBM Content Manager
• IBM DB2
• IBM FileNet P8
• IBM Lotus Notes
• IBM WebSphere
• iManage Work
• Jive
• LegalKEY
• LexisNexis Interaction
• Lotus Notes Databases
• Microsoft Dynamics CRM
• Microsoft Exchange
• Microsoft Exchange Public Folders
• Microsoft SQL Server
• MySQL
• NetDocuments
• Neudesic The Firm Directory
• Objective
• OpenText LiveLink/RM
• OpenText eDOCS DM
• Oracle Database
• Oracle WebCenter
• Oracle WebCenter Content (UCM/Stellent)
• PLC/Practical Law
• ProLaw
• Salesforce.com
• SAP ERP
• ServiceNow
• SharePoint Online
• SharePoint 2016
• SharePoint 2013
• SharePoint 2010
• SharePoint 2007
• Sitecore
• Any SQL-based CRM system
• Veeva Vault
• Veritas Enterprise Vault
(Symantec eVault)
• West km
• Xerox DocuShare
• Yammer
Seamless Connectivity
Single point of access
–
–
–
–
Secure Connectivity to Enterprise Systems
Provide unified views of all knowledge assets
–
–
–
–
–
–
Metadata Drives Great User Experiences
Documents from many sourcesAll client or matter-relevant documents are integrated.
Rich MetaDataContent annotated automatically – concepts,
categories, citations, matters, clients, etc
Navigation ControlsExplore, Discover, Drill-down
Manual Tagging is impractical
and remarkably inconsistent
Automation
Called: AutoClassification, AutoTagging, Metadata Generation, Text Analytics, ….
Demo
34
The
Evolution
of
SharePoint:
HYBRID Management ExtensibilityExperiences
| Server
Experiences Management Extensibility
| Server | Server
HYBRID
Team
Sites
Portals
Enterprise
Content Mngt
BI
Cloud Hybrid Search
Demo
42
SharePoint 2013/2016 Search Architecture
Web Service (CEWS)
Reduce your footprint
Servers
Volume of Content(indexable items) Pattern
On-prem Search Farm
Cloud Hybrid Search
0-10 million items small 4 App + 2 DB 1 or 2
10-40 million items medium 12 App + 2 DB 2
40-100 million items large 28 App + 4 DB 2
400 million items XL example (SP2016) 86 App + 4DB 2 or 3
SharePoint Server
(On-premises or Hosted)Office 365
SharePoint Online Content
Onedrive for Business ContentSharePoint Content
Cloud Hybrid Search
Case Study: Split Users with SharePoint
SupportSales & Marketing
Knowledge Articles
Fileshares
OneDrive
Support forum
SPO
Search Farm
SP 2013 content SP 2010 content
On-premises
Office 365
SPO content
SP 2013/2016
Cloud SSA
External Content
(on-premises and/or
in the cloud)
SharePoint Server
(On-premises or Hosted)Office 365
SharePoint Online Content
Onedrive for Business Content
Co
nnect
ors
SharePoint Content
Adding External Content
Cloud Hybrid Search
Also drives:
• Office Graph (delve,..)
• Compliance (DLP, …)
External Content in O365 UX
Unified view across all content
- on-premises and on-line
- inside and outside SharePoint
Current Caveats:
1) don’t see thumbnails, just file icons
2) Have to query for it to show up
–
–
–
–
Case Study: Cloud SSA, external content
Large global company
in materials science
Issues with Cloud Hybrid Search (1)Cloud Hybrid Search "annoyances"
Performance Characteristicsslower query latency for on-prem queries against Cloud SSA
SharePoint Online Limitationsno synonyms
no site-level schema
no full trust code access
Hybrid Administration Weaknessesclunky metadata mapping
can't remove on-premises search results from Cloud SSA
trickier to test & debug crawls
can't reset index from Cloud SSA
Be aware of these
& compensate for them
(Fixed in August PU)
(Semi-addressed in June PU)
And it’s getting better:
Issues with Cloud Hybrid Search (2)
54
Content Enrichmentno CEWS
no Entity Extraction
Securityno Custom Security Trimming
Can't crawl across Multiple Domains
Can't Crawl SP in Classic Auth Mode
Data Sovereigntyexport-restricted content
can't be put in O365 index
Limitations of Cloud SSA
External Content
(on-premises and/or
in the cloud)
SharePoint Server
(On-premises or Hosted)
SPO Content
OneDrive Content
Co
nnect
ors SharePoint Content
Connector
Framework
Office 365
AutoClassifier
(app version)
CEWS
Custom
Processing
Case study:Content Enrichment
Content
CloudSSA
Connector Framework
IndexingConnectors
Smart Pipeline
AutoClassifierCustom Stage A
CustomStage C
Custom Stage B
Case Study: Crawling Cross-Domain
A global single index solution
Cloud SSA
Cloud SSA
Cloud SSA
Cloud SSA
Cloud SSA
BUT export-restricted content
can’t be in the global index
Issues with Cloud Hybrid Search OOB
Content Enrichmentno CEWS
no Entity Extraction
Securityno Custom Security Trimming
Can't crawl across Multiple Domains
Can't Crawl SP in Classic Auth Mode
Data Sovereigntyexport-restricted content
can't be put in O365 index
Limitations of Cloud SSA BA Insight Solution
Connector Framework
AutoClassifier
Connector Framework
can 'map down' to AD groups
can 'map across' cross-domain
can crawl and map security
Federator
A new wave of Intelligent Search / Intranets?
Relevant: Deliver all content users need, independent of where it resides
Proactive: Intelligently and automatically deliver content in context
Interactive: Present users with familiar search patterns
Pillars of intelligent search
All of these:
• Depend on being Connected, Structured, Personalized, & Adaptive
• Use NLP and ML in addition to traditional search technologies
–
–
–
–
–
Turning Microsoft Search into an Insight Engine
Demo
66
Office 365USER
EXPERIENCES
Unified Search IndexSEARCH INDEX
Office 365DATA
REPOSITORIES
Search & discovery architecture --Online, on-premises, and hybrid
Office 365MICROSOFT
GRAPH
Cloud-hosted, on-premises &
mobile apps
On-premises
Content
ContentProcessing
SearchIndex
Search UI
ML with BA Insight
QueryProcessing
AutoClassifierFlexible combination of Rules & ML
FederatorPluggable query and results pipelines
Analytics
Smart AnalyticsBehavioral data for insight and learning
–
What is Natural Language Processing?
Complicators
–
–
–
–
–
Content
ContentProcessing
SearchIndex
Search UI
NLP with BA Insight
QueryProcessing
AutoClassifierFlexible combination of Rules & ML
FederatorPluggable query and results pipelines
Analytics
Smart AnalyticsBehavioral data for insight and learning
AutoClassifierrules-driven core, combined with learning-based modules
Enhanced Content
Enriched with
Metadata and
Content Types
Search Visualization Workflow
Federator: pluggable query & results processing
•
•
•
•
Content
ContentProcessing
SearchIndex
Search UI
ML applied to search
QueryProcessing
Learning about content structure Learning about user intent
Analytics
Behavioral data to drive learning
Learning to rank
79
Search practices are improving
Why Can’t Everything in an Intranet be Dynamic?
• News
• Forums
• Content
• Navigation
• Tools/Applets
• Analytics IT CAN
Nearly every Intranet will need to add
1. ConnectivityConnectors to content & systems outside of O365
3. ContextPersonalization through behavioural information
2. Structure Information Architecture & Tags for navigation &
findability
Smart Content needs Gardeners
