apache solr changes the way you build sites

APACHE SOLR CHANGES THE WAY YOU BUILD SITES

How to build dynamic navigation for dynamic content

Jacob Singh and Peter Wolanin Drupalcon Paris, September 3rd 2009

RELEVANT AND FAST

FILTERS

SORTABLE

MORE LIKE THIS

SITE NAVIGATION

INFORMATION ARCHITECTURE

•Is the science and art of guessing what your users want to see or do on your site and helping them get there

•Often done without actually consulting visitors or proper understanding of the target market

ARCHETYPES - WHO

CONTENT - YOU GOT IT

ORGANIZE AND PROSPER!

NAVIGATION

SO? WHAT’S WRONG WITH THAT?

Site Map!

(1996)

WEBSITES GROW.

•Sometimes really fast.

• And always in ways you didn’t expect

WHY DOES THIS HAPPEN?

VISITORS ARE TOO VARIED TO THINK OF THEM ALL

YOUR CONTENT IS TOO VARIED FOR YOUR MENU

WHERE ARE BEER HATS?

YOUR CONTENT IS TOO VAST AND TAKES TOO MANY

CLICKS TO FIND

TOOK 7 CLICKS TO FIND DRUPAL

YOUR CONTENT IS CONTRIBUTED BY USERS

AND UNPREDICTABLE.

DRUPAL IS A SOCIAL PUBLISHING PLATFORM

YOU USE IT SO USERS WILL ADD CONTENT

That’s the point!

IF WE HAVE DYNAMIC USER CONTRIBUTED CONTENT

WHY DO WE INSIST ON STATIC ADMIN DEFINED

NAVIGATION?

IF WE HAVE DYNAMIC USER CONTRIBUTED CONTENT

WEB 2.0 JARGON TO SAVE THE DAY

•Tag Clouds

•Content Recommendation

•Social Networking

•Social Bookmarking

HOW SEARCH DIED AND

HOW WE BRING IT BACK

! ?

SEARCH IS NOT A PRIORITY

MOST DRUPAL WEBSITES REALLY IGNORE SEARCH

CAN YOU FIND THE SEARCH BOX?

HINT: I’VE CIRCLED IT IN ORANGE

WHAT DO THE BIG SITES DO?

WHY DID SEARCH DIE?

•It was too slow

•It wasn’t smart enough

•Users learned not to trust it

LANGUAGE IS IMPORTANT

GOLDEN RULE: No Dead Ends!

SEARCH ON G.D.O

FIND OUT WHERE TO LOOK

FOUND ‘EM

I LEFT MY TIME MACHINE AT HOME

SMARTER MATCHES

STEMMING

SPELLING SUGGESTIONS

BUT I’M THE ADMIN!

ALL THE COOL KIDS ARE DOING IT

DAMZ EN L’MASSION

A TAILOR MADE NAVIGATION FOR EVERY USER

You want it, right?

The Apache Solr Project

• Stable and proven.– Used by Netflix, CNET, CitySearch, StubHub!, GameSpot, AOL– Full time maintainers– VERY Active mailing list (about 1k messages per month)

• Fast: written in Java.• Uses Lucene: the top open source search

library.• Distibuted: scales out in multiple directions.

Apache Solr Search Integration

• Very active project on drupal.org.• Takes advantage of latest Solr features.• Exposes an API to modify search and display

behavior.• Supported by engineers at Acquia.• All Acquia code improvements have been

contributed back to the Drupal.org project.• Many of the Drupalcon sponsors and attendees

are already involved and using it.

Feaure highlights

• Taxonomy, user, and language facets.• Node type faceting, weighting, and exclusion.• Node property (e.g. sticky) and date weighting.• Date facets on content creation or change.• OG facets (optional sub-module).• Node access respected (optional sub-module).• More-like-this content recommendations.• Customizable (see drupal.org module browsing

features).

Integration with other modules/data

• Webmail: http://drupal.org/project/webmail_plus

• Attachments: http://drupal.org/project/apachesolr_attachments

• Views: http://drupal.org/project/apachesolr_views

• Services: http://drupal.org/project/solr_service

• Nodequeue: http://drupal.org/project/nodequeue

• RDF: http://drupal.org/project/apachesolr_rdf

• Project: http://drupal.org/project/project

• Ubercart: http://drupal.org/project/apachesolr_ubercart

You Can Run Yourself. Easy!1. Get a dedicated server or a VPS and get Solr loaded on it.

2. Find a Java server administrator or get some books.

3. Get the Drupal module, install the PHP library, and configure it.

4. Replace the stock Solr configuration files with Drupal ones.

5. Learn about Solr replication and configuring it.

6. Set up log management, alerting, monitoring, etc.

7. Implement regular upgrades or patches to Solr which will requiring getting your Java development set up and building from source sometimes.

8. Keep up to date with the Drupal module.

9. Implement a security regime to protect data transfer (i.e. so spammers can’t add Viagra ads to your search results)

10. Harden your servers, setup firewalls and IP-based, password-based, or other security.

11. Figure out how handle updates and versioning of Solr and your schema.

12. *Recommended: Get on the solr-user and solr-developer mailing lists to get updates and alerts on the Apache Solr project. Don’t worry, it’s only a 50 or so mails a day if you don’t count the commit messages.

Or... use Acquia Search

• Sign up on acquia.com.• Free 30 day trial subscriptions for anyone.• You must be running a Drupal 6.x site, with PHP

5.2.0+ (5.1.4+ possible as well).• Use Acquia Drupal or install our search module

package.• It leaves Drupal core search intact, so you can

go back anytime.• Convert your site and start impressing users!• We will worry about everything else.

How Acquia Search Works

Your webserver

Search slave servers

Search master server

AcquiaNetwork

content to index

SSL, HMAC

authenticatedrequest

indexreplication

search request

SSL, HMACauthenticated

results

SSL, HMAC

Proving the platform

• Benchmarking our servers, on the search server itself, most searches run in < 200 ms, even under high load.

Who Is This For?

• Small and medium size sites - easy access to enterprise search for every Drupal site.– No hardware, no experience, fast setup, low cost.

• Large sites and Acquia partners - the same solution you’d deploy, but faster and easier.– Don’t consume your engineering resources.– Why load your own servers? – We handle the security and availability.– Impress your users and clients.

How-to Screencasts

• http://www.jeffnoyes.com/content/how-use-acquia-search-apache-solr

• http://www.jeffnoyes.com/content/enabling-acquia-search-and-apache-solr

Here’s a quick look at the admin interface:

© 2008 Acquia, Inc.

New modules to enhance the experience

© 2008 Acquia, Inc.

Searching with Views 3• Define filters, sorts in Views like normal• Solr, not MySQL, gets the query• Views is responsible for rendering

results:• you configure the visible fields• grid views• carousels • search results in blocks

• Faceting works like in current Apache

© 2008 Acquia, Inc.

Javascript interface• http://drupal.org/project/solrjs

© 2008 Acquia, Inc.

Javascript interface• http://drupal.org/project/solrjs

© 2008 Acquia, Inc.

Autocompletehttp://drupal.org/project/apachesolr_autocomplete

© 2008 Acquia, Inc.

Stats for administratorshttp://drupal.org/project/apachesolr_stats

© 2008 Acquia, Inc.

DRUPAL-6--2 Branch

© 2008 Acquia, Inc.

DRUPAL-6--2 Branch

• Researching more substantial architectural changes to query building.

• Looking at ways to support multi-site and better support multi-lingual content.

Learn and Contribute• Find us at the code sprint (Saturday) if you have

questions about the code or roadmap.• Come to the Acquia table any time to learn

more about Acquia Search.

jacob.singh@acquia.competer.wolanin@acquia.com

Thank you!

© 2009, Acquia, Inc.