rendre ajax crawlable par les moteurs
TRANSCRIPT
![Page 1: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/1.jpg)
Making AJAX crawlable
Katharina ProbstEngineer, GoogleBruce JohnsonEngineering Manager, Google
in collaboration with:Arup Mukherjee, Erik van der Poel, Li Xiao, Google
![Page 2: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/2.jpg)
The problem of AJAX for web crawlers
Web crawlers don't always see what the user sees• JavaScript produces dynamic content that is not seen by crawlers• Example: A Google Web Toolkit application that looks like this to a user...
...but a web crawler only sees this: <script src='showcase.js'></script>
![Page 3: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/3.jpg)
Why does this problem need to be solved?
• Web 2.0: More content on the web is created dynamically (~69%)• Over time, this hurts search• Developers are discouraged from building dynamic apps
• Not solving AJAX crawlability holds back progress on the web!
![Page 4: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/4.jpg)
A crawler's view of the web - with and without AJAX
![Page 5: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/5.jpg)
• Crawling and indexing AJAX is needed for users and developers
• Problem: Which AJAX states can be indexed?o Explicit opt-in needed by the web server
• Problem: Don't want to cloako Users and search engine crawlers need to see the same content
• Problem: How could the logistics work?o That's the remainder of the presentation
Goal: crawl and index AJAX
![Page 6: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/6.jpg)
Possible solutions
• Crawlers execute all the web's JavaScripto This is expensive and time-consuming o Only major search engines would even be able to do this, and
probably only partiallyo Indexes would be more stale, resulting in worse search results
• Web servers execute their own JavaScript at crawl timeo Avoids above problemso Gives more control to webmasters o Can be done automaticallyo Does not require ongoing maintenance
![Page 7: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/7.jpg)
Overview of proposed approach - crawl time
Crawling is enabled by mapping between • "pretty" URLs: www.example.com/page?query#!mystate• "ugly" URLs: www.example.com/page?query&_escaped_fragment_=mystate
![Page 8: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/8.jpg)
Overview of proposed approach - search time
Nothing changes!
![Page 9: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/9.jpg)
Agreement between participants
• Web servers agree too opt in by indicating indexable states o execute JavaScript for ugly URLs (no user agent sniffing!) o not cloak by always giving same content to browser and crawler
regardless of request (or risk elimination, as before) • Search engines agree to
o discover URLs as before (Sitemaps, hyperlinks) o modify pretty URLs to ugly URLso index contento display pretty URLs
![Page 10: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/10.jpg)
Summary: Life of a URL
http://example.com/stocks.html#GOOG
could easily be changed to
http://example.com/stocks.html#!GOOG
which can be crawled as http://example.com/stocks.html?_escaped_fragment_=GOOG
but will be displayed in the search results as
http://example.com/stocks.html#!GOOG
![Page 11: rendre AJAX crawlable par les moteurs](https://reader036.vdocuments.site/reader036/viewer/2022082920/5555d52ad8b42aaf158b4eef/html5/thumbnails/11.jpg)
Feedback is welcome
• We are currently working on a proposal and prototype implementation• Check out the blog post on the Google Webmaster Central Blog:
http://googlewebmastercentral.blogspot.com• We welcome feedback from the community at the Google Webmaster
Help Forum (link is posted in the blog entry)