building specialized industry apps using solr - by rahul agarwalla
DESCRIPTION
See conference video - http://www.lucidimagination.com/devzone/events/conferences/revolution/2011TRANSCRIPT
©2011 Uchida Spectrum, Inc. All rights reserved.
Building specialized applica/ons using Solr; Migra/on from FAST ESP
Rahul Agarwalla Head of Interna/onal Business
Uchida Spectrum Inc.
©2011 Uchida Spectrum, Inc. All rights reserved. Page-2 Page-2
1995 ~ • So)ware License Sales • License Management Repor:ng • License Procurement System • License Adjustment Consul:ng
1997 ~ • Network System Consul:ng Services
― Ac:ve Directory Network ― Exchange Messaging Network
• License Management System Consul:ng ― So)ware Management Server
• Portal System Consul:ng ― Share Point Portal Server ― Websphere Portal Server
2002 ~ • Enterprise Intelligence Applica:on
― SMART InSight G2 Enterprise ― SMART InSight G2 Professional
• Search PlaRorm Consul:ng & Support ― FAST ESP ― Lucene/Solr ― Lucid Works Enterprise
Enterprise Search Business
SoDware License Business
Network Technology Services
Uchida Spectrum Overview
©2011 Uchida Spectrum, Inc. All rights reserved. Page-3 Page-3
Some of Uchida Spectrum’s customers
©2011 Uchida Spectrum, Inc. All rights reserved. Page-4 Page-4
SMART/InSight History
2003: FAST Alliance
2004: PlaRorm for custom solu:ons
2005: SMART InSight 1.1
Customers in Japan, China & India: • 2 of top 3 Japanese car manufacturers • Top consumer electronics company • Large financial ins8tu8ons • China’s biggest eCommerce firm
©2011 Uchida Spectrum, Inc. All rights reserved. Page-5 Page-5
What is today’s buzz word?
Smart Phone
• Extreme scalability • Flexibility & Extensibility • Feature rich search
©2011 Uchida Spectrum, Inc. All rights reserved. Page-6 Page-6
What I learnt from the Japan catastrophe
©2011 Uchida Spectrum, Inc. All rights reserved. Page-7 Page-7
The power of community
Japanese People
[Open community]
• Quick response
• Disclose / Share • Prac:cal Impact
Japanese Government
[Closed/big brother]
• Slow, behind the curve
• Legacy/CYA • Confusion
Power shift Driver of innova/on
©2011 Uchida Spectrum, Inc. All rights reserved. Page-8
Lessons from FAST ESP Migra/on: advantage LWE/Solr
• Key Issues: 1. Smaller record and index size enable faster index maintenance
2. # of records per node: rule of thumb 10m vs. 2m 3. Licensing & Maintenance Cost: less than ½
• Scalability: 5x
• Cost Performance: 10x • High Flexibility
• Lower Opera/ons Cost • Faster Innova/on
©2011 Uchida Spectrum, Inc. All rights reserved. Page-9 Page-9
Enterprise Search expecta/ons
• Big data scale • Security is important • Disparate data: geography, systems, languages, format, structures
• KM is good to have, databases are cri:cal
• Support different users & usage: department, role, tasks
• High recall
©2011 Uchida Spectrum, Inc. All rights reserved. Page-10
Lessons from FAST ESP Migra/on: Filling the gaps
• Security • ACL security: complex requirements
• File System: file & folder level control • CRM/ERP… : Keeping ACLs up-‐to-‐date
• Content aggrega/on • Connectors • Normaliza:on
• Open source op:ons for ESP pipeline • Openpipeline
• Pypes
©2011 Uchida Spectrum, Inc. All rights reserved. Page-11
Building specialized applica/ons: Content fusion
• Content fusion from disparate data:
• Single index ≠ integra:on • Modeling of content rela:onships is essen:al
©2011 Uchida Spectrum, Inc. All rights reserved. Page-12
Virtual integra/on based on search
Search Index
Content store Big table, flat index
Applica/on layer Content sets and inter-‐rela/onships
Search Index Search Index
©2011 Uchida Spectrum, Inc. All rights reserved. Page-13
Virtual integra/on based on search…2
Search Index LWE Solr
LWE Adapter SolrAdapter
Result Pipeline
Que
ry Pipeline
Transform
Boos:ng
. . . . . .
App
end Pipe
line
Other ……
Tagging Pipe
line
Content Security
……
Search Service • Data transforma:on: - key:key, key:value, field names
• Query & Result transforma:on • Boos:ng / Relevancy algorithm • Security • Mul:-‐Language support • Federa:on & mashups
©2011 Uchida Spectrum, Inc. All rights reserved. Page-14
Building specialized applica/ons: Personaliza/on
• Applica/on flow depends on the task • Data Personaliza/on increases produc/vity
• SMART InSight approach: Task based UI • Schema independent widgets for analy:cs &
visualiza:on
• Portalized
• Personalized: widgets, func:ons, content, fields
©2011 Uchida Spectrum, Inc. All rights reserved. Page-15
Knowledge Center: made possible by Solr
Scalability and low TCO gives us ability to build new features • Knowledge Centre has logs of all user ac:vity in SMART InSight
• This would be too costly with a commercial Search Engine and would not be feasible in a database
Using this rich data we can: • Profile users, groups and networks • Personalize Recommenda:ons
• Create social ranking algorithms
• Usage analy:cs
©2011 Uchida Spectrum, Inc. All rights reserved. Page-16
Widgets Personaliza/on Page Ajax Portal
Convergent Knowledge Framework
Virtual Integra/on Framework
Management & Security
Data Chain
SA
SA
SA
SA
Recommend
Profiling
Metadata Analysis
Knowledge Centre
Knowledge Log
Contents Set
Design
Engineering
Claims
Internal
Internet
NHTSA
Specs
PLM
Repair
CAD
EDR
Dealers
Claims
Overview of SMART InSight for Automo/ve
Task based UIs
Content Model
Early Defect Warning
Parts Catalog
Claim Analysis
Benchmarking
©2011 Uchida Spectrum, Inc. All rights reserved. Page-17
: Interac/ve Click Log Analysis System
Hadoop
Access Log
Solr
• > $50 Billion sales / year • > 800 Million Items • > 370 Million Users • Billions of clicks per day
xxxxxxxx
Xxxx Xxxx Xxxx Xxxx xxxx
Solr, Hadoop + SMART/InSight G2
©2011 Uchida Spectrum, Inc. All rights reserved. Page-18
: Global Research Community
Solr
• > 270 content sources: Socie/es, Associa/ons, Publishers & Open • IEEE, ACM… • Elsevier, Wiley, Springer…
Broadcast Search
Real /me indexing Dynamic Result Merging
• Top Academic Ins/tutes: • Faculty, Research Fellows & Post graduate students
• Govt. Departments & Corporate R&D • Scien:sts and researchers
Research Discovery & Collec/ve Intelligence (Knowledge Centre)
©2011 Uchida Spectrum, Inc. All rights reserved. Page-19
Demonstra/on
©2011 Uchida Spectrum, Inc. All rights reserved. Page-20
Contact Details
Rahul Agarwalla Head – Interna/onal Business
www.spectrum.co.jp