search engine - how to make it

Search EngineHow To Make it

Wednesday, December 12, 12

Search Engine

All documents

retrieved documents (RET)

relevant documents (REL)

RET ∩ REL

database search:- low recall- high precision

web search:- high recall- low precision

Search Quality Measurement


Search EngineFile

System

3rd party apps

Database

File System Crawler

Crawler API

Database Crawler

AaBb

Text Parser

HTML Parser

PDF Parser

AaBbPDFTextHTML

DocumentImage...

Document Enhancing

Documents (title,

summary, author,

datetime)

Indexer

Documents (Categorized, Taxonomized)

Stop AnalyzerLanguage Analyzer

Index Searcher Index

Mobile Client

Web Client

Index Searcher

Document Landing Page


Search Engine

• Process in Search Engine

• Crawling

• Parsing

• Indexing

• Searching


Search Engine• Process in Search Engine

• Crawling

• Parsing

• Duplicate Content Detection

• Document Enhancement

• Indexing

• Searching

• Document ServingWednesday, December 12, 12

Search Engine

• Crawling

• Collecting Data

• Input : Data content to Search

• Output : Raw Content Data in its original format


Search Engine• Crawling

AaBb

File System

3rd party apps

Database

File System Crawler

Crawler API

Database Crawler

AaBbPDFTextHTML

DocumentImage...


Search Engine

• Parsing

• Process to extract elements from crawled documents

• Input : Raw Contents

• Output : Textual Structured Documents


Search Engine• Parsing

AaBb

Text Parser

HTML Parser

PDF Parser

AaBbPDFTextHTML

DocumentImage...

Documents (title,

summary, author,

datetime)


Search Engine

• Content Duplication Detection

• Bigger Data means Bigger Duplication on Data

• Search Engine implement similiar document detection


Search Engine• Document Representation

Model: Term Frequency(Tf)Contoh:

Document 1(d1)=”andi likes to watch movie. His wife likes it too”

Document 2(d2)=”andi also likes to watch soccer game.”

Dictionary={1:andi, 2:likes, 3:watch, 4:movie, 5:wife, 6:too, 7:soccer}

Document representation in model Tf:d1={1, 2, 2, 2, 1, 1, 0}

d2={1, 1, 1, 0, 0, 0, 1}


Search Engine• Document Similiarity

Similarity between document d1 dan d2 : S(d1, d2)

S(d1, d2)=|d1-d2|

d1={1, 2, 2, 2, 1, 1, 0}

d2={1, 1, 1, 0, 0, 0, 1}

Contoh:

S(d1, d2)=|1-1|+|2-1|+|2-1|+|2-0|+|1-0|+|1-0|+|0-1|

S(d1, d2)=7

With above definition, less value we got means more those two documents are getting more similiar


Search Engine• Alghoritms

1. Counting Tf for every document

2. Find the smallest value of S(d, di) from all documents collection to get the most similiar of document d3. if the value of S(d, di) < threshold then document d and compared with create date, then erase older document4. Repeat process 2 dan 3 until there is no value of S that less than Theshold


Search Engine

• Document Enhancement

• Give tagging based on taxonomy


Search Engine• Document Enhancement

Document Enhancing

Documents (title,

summary, author,

datetime)

Documents (Categorized, Taxonomized)


Search Engine

• Indexing

• Indexing process from all information that have been gathered in one document

• Faster Searching process

• Able to search based on certain field


Search Engine• Indexing

IndexerDocuments

(Categorized, Taxonomized)

Index

Stop Analyzer

Language Analyzer


Search Engine

• Searching

Index SearcherIndex

Mobile Client

Web Client


Search Engine

• Document Serving

• Search Engine also has a function to display result


Search Engine

Index SearcherIndex

Mobile Client

Web ClientIndex

SearcherDocument

Landing Page


Search Engine• Recommended Open Source

Technology• Search Engine : Lucene, Nutch

• Programming Library : Hadoop, Scala Actor

• Database : MongoDB, PostgreSQL

• Programming Language : Java, Scala, PHP


Thank You
