an empirical study on selective sampling in active...

An Empirical Study on Selective Samplingin Active Learning for Splog Detection

Taichi Katayama1

Takehito Utsuro1

Yuuki Sato2

Takayuki Yoshinaka3

Yasuhide Kawada4

Tomohiro Fukuhara5

1University of Tsukuba, 2Konami Corporation, 3Tokyo Denki University,

4Navix Co., Ltd., 5University of Tokyo,

AIRWeb2009, April 21nd, 2009 @Madrid, Spain. WWW2009

Background

• Opinion Mining from Blogs

• Splogs are Serious Noise in Opinion Mining– e.g., larger scale statistics (2008 Mar.)

• 40% of Japanese Blog Articles in BuzzPulse, nifty are Splogs, 2007 Oct. � 2008 Feb.

• Automatic Detection is highly Expected.

keyword stuffed blog

Rumor of “FC Tokyo”(a football

team in Japan)

“FC Tokyo”

Blog snippet retrieved with

“FC Tokyo”

Blog snippet retrieved with

“LOUIS VUITTON Key case”

pop-up advertisement automatically inserted by the blog host system

$50 Software Package for Massive Splog Creation

Featuring• SEO• Affiliate Program

in link in link

satellite

satellite satellite

satellite

main site

Background

• Opinion Mining from Blogs

• Splogs are Serious Noise in Opinion Mining– e.g., larger scale statistics (2008 Mar.)

• 40% of Japanese Blog Articles in BuzzPulse, nifty are Splogs, 2007 Oct. � 2008 Feb.

• Automatic Detection is highly Expected.

Previous studies on splog detection

• [P.Kolari 2007]– Words– URLs– Anchor texts– Links – HTML meta tags

• [Y.-R.Lin 2007]– Temporal self similarities of

• Posting time• Posting contents• Affiliated links

• [G.Mishne 2005]– Language models among the blog post , the comment ,and

pages linked by the comments

Evaluation with two data sets“Does splog change over time?”

1. Years 2007-2008 (720 sites)2. Years 2008-2009 (720 sites)

��

� ��

Recall(%)

��

� ��

Recall(%)

Recall/Precision curves with confidence measureTrain 07-08(720

sites)

Train 07-08 (360 sites) +08-09 (360

sites)

Train 07-08 (360 sites) +08-09 (360

sites)

Train 07-08(720 sites)

Splog detection Authentic blog detection

Test 08-09 (40 sites)

Purpose of This Research (1)

• Needs for continuously updating splog/authentic blog data setsyear by year

• How to reduce human supervision?

• May active learning framework work?

Purpose of This Research (2)

• Optimal Strategies for Selective Sampling in Active Learning

• Guided by Certain Confidence Measure

random samples,

samples balanced with a

confidence measure

samples with theleast confidence

Outline

1. Definition of splog sites2. Splog detection by Machine learning

– SVM– Confidence Measure– Features

3. Active learning4. Evaluation5. Future works

Definition of splog sites• If one of the followings holds for the given

blog sites, then it is mostly splog– originally written text is not included– originally written text is included but many

• “links top affiliated sites” or• ”advertisement articles” or• “articles with adult content”

are included (judged individually by considering the contents of each blog)

• Otherwise, the given blog sites is an authentic blog

Splog Detection by SVMs

• a tool – TinySVM

• the kernel function:– 2nd order linear

• confidence measure – the distance from the separating hyperplane

to each test instance

A Confidence Measure

��

Lower Bound (authentic blog)

��

Separatinghyperplane

Lower Bound (splog)

�:splog�:authentic blog

Features for splog detection

1. Total frequency of URLs not linked from splogs2. Co-occurrence between Noun Phrases and

Splogs• Sum of

3. Noun Phrases in Anchor Texts and linked URLs• Total frequency of anchor text noun phrases

• in splogs• out-linked to splog URLs and Blacklist URLs

• Total frequency of anchor text noun phrases• in splogs• out-linked to authentic blog URLs Whitelist URLs

)phrasenoun,splog(2 w�

Feature1: URLs are not linked from splog

splogAuthentic

blogAuthentic

blogsplog splog

more than oneinward links

from authentic blogs

included only in splogs

included only in authentic blogs

Whitelist:defined as

these URLs

Blacklist:defined as

these URLs

Value of the Whitelist URLs feature

��

instance testthe

inoffrequencytotal

homepagesblogauthenticof

instancestraining wholein theof

frequencytotal

u: Whitelist URLs

Splogs• Sum of

Feature2: Noun Phrases

Training set

�� w �� w ��

splog Authenticblog

�� w ��

w: a noun phrase

freq(splog,w)=a freq(splog,�w)=b

freq(authentic blog,w)=c freq(authentic blog,�w)=d

�� w �� w �� w ��

�� w �� w ��

Value of the splog noun phrase feature

��

� instance test in theoffrequencytotal

),splog(log

))()()(()(),splog(

dcdbcababcadw

Splogs• Sum of

Feature3:Noun Phrases in Anchor Texts and linked URLs

a Splog site s

Blacklist URLs Splog URLsWhitelist URLs Authentic blog

http://��http://��http://��

AncfB(w,s)=freq of w

w: a noun phrase in Anchor text

AncfW(w,s)=freq of w

Other URLs

Noun Phrases in Anchor Texts and linked URLs: two features

� � ��

��

w stwAncfBswAncfB ),(),(log

� � ��

��

wtwAncfWswAncfW ),(),(log

homepagessplogtraining

w: noun phrases: a training splog homepaget: a test instance blog homepage

the value of a feature named anchor text noun phrase out-linked to Whitelist URLsfor a test instance blog homepage

the value of a feature named anchor text noun phrase out-linked to Blacklist URLsfor a test instance blog homepage

Framework of Active learning

Pool of unlabeledinstances

(initial size of 3504)

(1296 splog and2208 authentic

TrainingSet

(initial sizeof 10)

(4 splog and6 authentic

selectivesamplingIn activelearning

Trainingan SVM

classifier

unlabeled4 sites

Humansupervision

labeled4 sites

250 cycles up to 1010 training instances

Statistics of Splog/Authentic Blogs Data Set

390424591445Years2008-2009

total# of authenticblogs

# of splogsData Sets

Strategies of selective sampling(1/2)

Low High

High/Low Balanced

��

Strategies of selective sampling(2/2)

Low-Sp/Au High-Au

High/Low-Au Balanced-Sp/Au

��

Outline

1. Definition of splog sites2. Splog detection by Machine learning

– SVM– Confidence Measure– Features

3. Active learning4. Evaluation5. Future works

Measure for Performance evaluationafter active learning cycles

• Recall/Precision– Splog detection

– Authentic blog detection is considered in a similar fashion

• “| Tr |= 3500”, “Random”– “| Tr |= 3500” indicates a classifier trained with the whole 3504

instances in the pool– “Random” indicates a classifier trained with randomly selected

training instances

|Ts(splog)||)Ts(LBDTs(splog)|

recall

|)Ts(LBD||)Ts(LBDTs(splog)|

precision

Lower Bound of the Confidence Measure

��

� �

��

Lower Bound (splog)�:splog

�:authentic blog

)( sLBDTsTs(splog): the set ofreference splog sites

Measure for Performance evaluationafter active learning cycles

• Recall/Precision– Splog detection

– Authentic blog detection is considered in a similar fashion

• “| Tr |= 3500”, “Random”– “| Tr |= 3500” indicates a classifier trained with the whole 3504

instances in the pool– “Random” indicates a classifier trained with randomly selected

training instances

|Ts(splog)||)Ts(LBDTs(splog)|

recall

|)Ts(LBD||)Ts(LBDTs(splog)|

precision

��

� ��

Recall(%)

High/Low-AuLow-Sp/Au

High-Au

Random

Balanced-Sp/Au

|Tr|=3500

Recall/precision curve of Splog detection

��

� ��

Recall(%)

High/Low-AuLow-Sp/Au

High-Au

RandomBalanced-Sp/Au

|Tr|=3500

Recall/precision curve of Authentic blog Detection

Evaluation results: comparison of strategies for selective sampling

|TR|=3500RandomHigh/Low

Blance

HighLow

Low Random

Previous studies of active learning for text classification tasks

Splog/authentic blog detection

Support Vectors• only the support vectors have effect on deciding the

position of the separating hyperplane• the number of support vectors can be regarded as the

complexity of the learning task

��

�Separatinghyperplane

�,��support vector

��

� ��

# of Training Instances

�High/Low-Au

�Low-Sp/Au

�High-Au

� Random

�Balanced-Sp/Au

Changes in # of Support Vectors

RandomHigh/LowBalance

Low High

Evaluation result: # of support vectors• The number of support vectors linearly

increases• Performance of splog/authentic blog

detection increase much more slowly • About 20% of training instances are

constantly selected as support vectors

• In this task, more effective features should be added.

��

� ��

High/Low-Au Low-Sp/Au

High-Au

Random

Balanced-Sp/Au

|Tr|=3500

Change in maximum precision with recall as 30 %

of Splog Detection

��

� ��

High/Low-Au

Low-Sp/Au

High-Au

Balanced-Sp/Au

|Tr|=3500

Change in maximum precision with recall as 30 %

of Authentic blog Detection

Evaluation result: # of support vectors• The number of support vectors linearly

increases• Performance of splog/authentic blog

detection increase much more slowly • About 20% of training instances are

constantly selected as support vectors

• In this task, more effective features should be added.

Future works

• Incorporating other features– Post time and intervals – Html structures

• Manual examination of support vectors

Thanks for your attention

an empirical study on selective sampling in active...

Documents

active frequency selective surface with wide

platinum-promoted ga/al2o3 as highly active, selective, and

a critical survey of empirical methods for evaluating...

active monitoring of selective laser melting process …

nc division of mental health, developmental disabilities...

zno as an active and selective catalyst for

an empirical study on selective sampling in active learning...

bending the q(x)’s: where actuarial and behavioral...

chapter - 4 communication...

supporting information active/selective pdzn catalyst...

cell biology lecture 3. function of plasma membrane...

antecedents of selective revealing: an empirical study in...

an investigation of active and selective oxygen in

selective service system€¦ · selective service is...

empirical comparison of active learning strategies for...

active learning coms 6998-4: learning and empirical...

a closed-loop selective harmonic compensation for active...

inhibitors of protein synthesis 1.30s directed:...

empirical evaluation of active learning techniques for...

evaluation of active management of the government …...