roland dunn - what do searchbots actually do? - ionsearch 2012

ionSearch 2012

Roland Dunn

Partner – Refined Practice

Searchbots: Lost Children ... or ... Hungry Psychopaths? What Do Searchbots Actually Do? (and why it matters)

© 2007-2012 Roland Dunn


Straightforward for humans and searchbots to find your content?



Do you provide what your potential customers want?



Do you provide what your potential customers want?

Do you stand out? Are you compelling, engaging, useful, unique, the best?

“Googlebot ... may be unable to completely index all the content on your site”


Doesn’t tell us what searchbots index


Webserver Logfiles ....


Over 2 months, 40% of all Googlebot’s requests were to …. just 2 URL’s (on-site search URLs)


Over 2 months, 40% of all Googlebot’s requests were to …. just 2 URL’s (on-site search URLs) On a website with approximately 40,000 URLs in Google’s index (allegedly, according to site:…)


Over 2 months, 40% of all Googlebot’s requests were to …. just 2 URL’s (on-site search URLs) On a website with approximately 40,000 URLs in Google’s index (allegedly, according to site:…) On a website serving approx. 150-200K unique natural search visits/month


Over 2 months, 40% of all Googlebot’s requests were to …. 2 URL’s. On a website with approximately 40,000 URLs in Google’s index. On a website serving approx. 150-200K unique natural search visits/month


Categorise URLs by “top level section” – perhaps more useful than just URLs ...


Table of “top level sections” requested by Googlebot



Chart of “top level sections” requested by Gbot – 6 months

Does Googlebot request all URLs served by a website?


Does Googlebot request all URLs served by a website?

No!



How Does Googlebot Spend Its Time? What Does it Request?

Does Googlebot spend its time cost-effectively?


Does Googlebot spend its time cost-effectively?


Not always.

Googlebot does not always request all content


Googlebot can become distracted, obsessed, or even lost:


Googlebot can become distracted, obsessed, or even lost:

• On-site search

• Additive filters/faceted URLs e.g. “shoes?size=3&colour=green&price=10&brand=smith”

• Sections with thin and/or very similar content


Googlebot may not spend its time efficiently

It needs help focusing on what we value


SMX West, March 2011, Matt Cutts: (http://goo.gl/ZZz7E):

“... if Google determines a site isn’t as useful to users, they may not crawl it as frequently"


http://goo.gl/ZZz7E

Suggestions:

• Embrace logfiles – full of useful information


Suggestions:


• Periodically check your searchbot behaviour


Suggestions:


• Periodically check your searchbot behaviour

• Especially if the “Googlebot found a high number” message keeps appearing


If searchbots appear distracted, unfocused, lost:



• Alter internal navigation & linking (e.g. flatten hierarchy)




• Robots.txt out (blunt approach)





• Alter URL construction (expensive!)





• Alter URL construction (expensive!)

• Check “sitemaps” – all URLs? What priorities expressed?


Lost Children ... or ... Hungry Psychopaths?


Or .... Distracted Teenager?



Sonification: What do Search Engines Requests Sound Like?

Online Branding:

http://www.refinedpractice.com/

T: @RefinedPractice


Natural Search:

http://www.cloudshapes.co.uk/

T: @roland_dunn

Slides Available At:

http://www.cloudshapes.co.uk/talks/

http://www.refinedpractice.com/

http://www.cloudshapes.co.uk/

http://www.cloudshapes.co.uk/talks/

roland dunn - what do searchbots actually do? - ionsearch 2012

Technology