structure based information extraction (sbie)

Structure Based Information Extraction (SBIE)

Hua Lei

06/09/2004

The disadvantages of BYU tool

1. Define ontologies, lexicons and data patterns for each domain.

2. Define and update ontologies, lexicons and data patterns manually.

3. Results heavily rely on lexicons and data patterns.

HTML code of a data group

<TD vAlign=top width="33%"><A class=prodtitle href="http://www.kmart.com/product/index.jsp?productId=1789425&amp;cp=784867.784872.784574&amp;parentPage=family"><CENTER><IMG src="kmart_files/p1472053th.gif" border=0></CENTER><BR>Sharp LC-20B4U-S 20-Inch Flact LCD Television in Silver</A><BR><BR><SPAN class=listprice>List Price: $1299.99* <BR></SPAN><SPAN class=ourprice><B>Our Price: $1199.99</B></SPAN> </TD>

Phenomenon: all data groups in the same web page have a same structure.

Idea: extracts data from web pages based on the data group structure.

Method of SBIE

Step 1. choose a data group as the initial one which has a typical data structure.

Brand Size Product No. Features Price

Sharp 20 LC-20B4U-S Flact LCD Television in Silver

$1199.99

• Recognize the annotations, data patterns and relative positions of each data.

• This algorithm can integrate all these structure information and get the structure of the data group.

• Validate the data group structure by recognizing other data groups based on this data group structure.

Method of SBIE

Step 2. Analyze HTML code of the web page and find the structure of data group.

<TD vAlign=top width="33%"><A class=prodtitle href="http://www.kmart.com/product/index.jsp?productId=1789425&amp;cp=784867.784872.784574&amp;parentPage=family"><CENTER><IMG src="kmart_files/p1472053th.gif" border=0></CENTER><BR>Sharp LC-20B4U-S 20-Inch Flact LCD Television in Silver</A><BR><BR><SPAN class=listprice>List Price: $1299.99* <BR></SPAN><SPAN class=ourprice><B>Our Price: $1199.99</B></SPAN> </TD>

Step 3. use the result of step 2 to extract other data groups in that web page.

Method of SBIE

Brand Size Product No. Features Price

Sharp 20 LC-20B4U-S Flact LCD Television in Silver $1199.99 JVC 20 Stereo TV/Full Dark Tint $169.99 Magnavox 20 20MT133S Color TV $149.99 Philips Magnavox

19 MT1905B Portable Color TV $139.99

RCA 19 F19421 TV $119.99 Sharp 19 19RM100 Color TV with Dark Tint Tube $119.99 Sylvania 19 TV $89.99

Method of SBIE

Initial instances

Structure analysis

Validation

No

Yes

Extract data

Machine Learning:

• A machine learning technique will combine with the structure recognizing algorithm.

• A database for data group structures, lexicons and data patterns will be created or update after each extraction.

• The machine learning tool can analyze new data group structure based on the structure information in the database.

• When SBIE is well trained, SBIE could analyze the data group structures in the HTML code and extract data without the initial data group (step 1).

Method of SBIE

•Extend BYU tool.

•Extract information without defining ontologies.

•Create and update data patterns and lexicons automatically.

•Extracting data based on their relative positions.

•Machine learning technique makes it smart.

Conclusion

SBIE

Questions