scalling solr in the cloud - oscon data 2011

www.nutshell.com

Scaling Solr in the CloudBy @ablyler & @LindsaySnider

F R O M

@NutshellCRM

http://www.nutshell.com


Agenda

About the Speakers

About Nutshell

Nutshell and Solr

Solr Resource Usage

Scaling Methods

Questions and Answers

About the Speakers

BitLeap / Barracuda NetworksDevelopers on the Backup Appliance / Cloud

Scaled databases and storage systems

Used SugarCRM & Salesforce.com

About Nutshell

Web and Mobile CRM application

Heavy use of OpenSource technologies:Gentoo

Nginx / PHP / ZendFramework / jQuery

MySQL / Solr / Gearman

Jenkins / Redmine / Cacti / Nagios

Nutshell and Solr

Heavy use of Solr for searching, table views, and de-duplication

Used for searching / display:Accounts, Competitors, Contacts, Leads, Products, Sources, Teams, and Users

Used for de-duplication:Accounts, Contacts

Reads vs Writes

0

1750

3500

5250

7000

Jul 20, 2011 3:00 AM Jul 20, 2011 6:00 AM Jul 20, 2011 9:00 AM Jul 20, 2011 12:00 PM Jul 20, 2011 3:00 PM Jul 20, 2011 6:00 PM Jul 20, 2011 9:00 PM Jul 21, 2011 12:00 AM

Writes Reads Time

Average Read Query Time: 2.11ms

Demo

www.nutshell.com



Computer Resources

CPU Disk I/O

RAM Disk Storage

CPU Disk I/O

RAM Disk Storage

SOLR Resources

CPU Disk I/O

RAM Disk Storage

Light

Heavy

SOLR Resources

CPU Disk I/O

RAM Disk Storage

Light

Heavy CPU

SOLR Resources

CPU Disk I/O

RAM Disk Storage

Light

Heavy CPU Disk I/O

SOLR Resources

CPU Disk I/O

RAM Disk Storage

Light

Heavy CPU Disk I/O

Disk Storage

SOLR Resources

CPU Disk I/O

RAM Disk Storage

Light

Heavy CPU Disk I/O

Disk Storage

RAM

Auto Provisioning

Setup new Solr Jetty app

Create MySQL database

Populate MySQL / Solr with demo data

Send welcome email

Dark Ages Age of Enlightenment TodayModern Era

Separate Jetty per Customer

Uses a ton of memory

Separate schema / Solr for each customer

Ran into upper limit morning before launch


Auto provisioning

Separate Jetty app for each customer


Auto provisioning

Separate Jetty app for each customer


0

25

50

Solr Core per Customer

Allows for management of Solr on a per customer basis: creating / stopping

Contained within a single Jetty app

Shared schema between all cores

Easily managed via simple HTTP API

Age of EnlightenmentDark Ages TodayModern Era

Fallback to MySQL

Landing page of application

Allows for graceful handling when Solr is down

Abstracted within the application library


Sun JVM to IcedTea JVM

IcedTea JVM uses less memory than Sun JVM


Separate Solr core per customer

Fallback to MySQL for table data

Migrated from Sun JVM to IcedTea JVM


Separate Solr core per customer

Fallback to MySQL for table data

Migrated from Sun JVM to IcedTea JVM

0

150

300


Shared Schema Across Cores

Decreases initialization time for each core

Decreases memory usage


Solr Index Field Selection

More indexed fields = more used memory

Only index fields that are searched

Store other non-indexed fields for display


Splitting of Reader / Writer

Index building is CPU / disk intensive

Writer = Solr with caching disabled

Reader = Solr slave, that doesn’t build indexes


MySQL(master-master)

Web

Solr Slave



Web

Solr Slave

Solr Master

Range indexerRange indexerRange indexerRange indexerDocument indexers

Gearman workers



Web

Solr Slave

Solr Master

Reindex ManagerRange indexerRange indexer

Range indexerRange indexerRange indexers



Web

Solr Slave

Solr Master

Reindex ManagerRange indexer

Range indexerRange indexerRange indexerRange indexers


Shared schema across cores

Solr index field selection

Splitting of reader / writer roles

Intelligent indexing / reindexing


Shared schema across cores

Solr index field selection

Splitting of reader / writer roles

Intelligent indexing / reindexing


0

500

1000

Intelligent Core Management

Least recently used Solr cores are spun down

Solr cores started on login

Reindexing is a database flag, and happens on the next login


Partitioning

Pairs of readers and writers

Partitioned based on account idLeft pad the account id w/ zeros to length of two

Reverse the account id

Take last two digits of the account id


Intelligent core spin down / up

Partitioning of customers to separate Solr servers


Intelligent core spin down / up

Partitioning of customers to separate Solr servers


0

4000

8000

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

123456

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

123456

1 2 3 4 5 6

1 2 3 4 5 6

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

123456

1 2 3 4 5 6

1 2 3 4 5 6

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

123456

1 2 3 4 5 6

65

1 2 3 4 5 6

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

123456

1 2 3 4 5 6

65

01

02

03

04

65

66

96

97

98

99

1 2 3 4 5 6

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

123456

1 2 3 4 5 6

65

01

02

03

04

65

66

96

97

98

99

65

1 2 3 4 5 6

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

123456

1 2 3 4 5 6

65

01

02

03

04

65

66

96

97

98

99

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

01

02

03

04

65

66

96

97

98

99

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

01

02

03

04

65

66

96

97

98

99

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

01

02

03

04

65

66

96

97

98

99

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

SOLR Server

8000 User

01

02

03

04

65

66

96

97

98

99

SOLR Server

8000 User

SOLR Server

8000 User

7000 Users

600 Users

300 Users

50 Usersauto provisioning

separate Jetty app for each customer

separate Solr core per customer

fall back to MySQL for table data

migrated from Sun JVM to IcedTea JVM

shared schema across cores

solr index field selection

splitting reader / writer roles

intelligent core spin down / up

parititioning of customer to separate Solr servers

Scaling Solr in the Cloud

By @ablyler & @LindsaySniderF R O M

@NutshellCRM

scalling solr in the cloud - oscon data 2011

Technology

solr cores

management of solr

solr slave

scaling solr

separate solr core

solrheavy use of solr

cpu disk io diskramstorage

icedtea jvm icedtea