evaluating the wisdom of crowds in assessing moore evaluating the wisdom of crowds in assessing...

Download Evaluating the Wisdom of Crowds in Assessing   Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites. Introduction to PhishTank data collection and analysis

Post on 25-Apr-2018

213 views

Category:

Documents

1 download

Embed Size (px)

TRANSCRIPT

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Evaluating the Wisdom of Crowds in AssessingPhishing Websites

    Tyler Moore and Richard Clayton

    University of CambridgeComputer Laboratory

    12th International Financial Cryptography and Data SecurityConference (FC08)

    Jan. 28, 2008, Cozumel Mexico

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Outline

    1 Introduction to PhishTank data collection and analysis

    2 Testing the accuracy of PhishTanks crowd decisions

    3 Disrupting PhishTanks verification system

    4 Comparing open and closed phishing feeds

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Outline

    1 Introduction to PhishTank data collection and analysis

    2 Testing the accuracy of PhishTanks crowd decisions

    3 Disrupting PhishTanks verification system

    4 Comparing open and closed phishing feeds

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    PhishTank

    Online community established in 2006 using the wisdom ofcrowds to fight phishing

    Users contribute in two ways1 Submit reports of suspected phishing sites2 Vote on whether others submissions are really phishing or not

    Data collection

    We examined reports from 200 908 phishing URLs submittedbetween February and September 2007For 24 254 reports, the site was removed before voting wascompleted, leaving 176 366 complete submissions3 798 users participated, casting 881511 votes= 53 submissions and 232 votes per user. But . . .

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Density of user submissions and votes

    1 100 10000

    15

    5050

    0

    # submissions per user

    Coun

    t

    1 100 10000

    15

    5050

    0

    # votes cast per userCo

    unt

    Top two submitters (93 588 and 31 910) are anti-phishingorganizations

    Some leading voters are PhishTank moderators the 25 moderators cast 74% of votes

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    User participation in PhishTank follows power law

    5e+01 5e+02 5e+03 5e+04

    0.00

    10.

    005

    0.05

    00.

    500

    Number of submissions x

    Prob

    (# s

    ubs

    > x)

    , us

    ers

    > 60

    sub

    s

    5e+01 5e+02 5e+03 5e+041e

    041e

    02

    1e+0

    0

    Number of votes cast xPr

    ob(#

    vot

    es >

    x),u

    sers

    > 3

    0 vo

    tes

    Power-law dist. Kolmogorov-Smirnov xmin D p-value

    Submissions 1.642 60 0.0533 0.9833Votes 1.646 30 0.0368 0.7608

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    User participation in PhishTank follows power law

    What does a power-law distribution mean in this context?

    A few highly-active users carry the loadMost users participate very little, but their aggregatedcontribution is substantial

    Why do we care?

    Power-law distributions appear often in real-world contexts,including many types of social interactionThis suggests skewed participation naturally occurs forcrowd-sourced applicationsPower laws invalidate Byzantine fault tolerance subvertingone highly active participant can undermine system

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Rock-phish attacks and duplicate submissions to

    PhishTank

    Rock-phish gang operate different to ordinary phishing sites1 Purchase several innocuous-sounding domains (lof80.info)2 Send out phishing email with URL

    http://www.volksbank.de.netw.oid3614061.lof80.info/vr

    3 Gang-hosted DNS server resolves domain to IP addressesof compromised machines that proxy to a back-end server

    Wildcard DNS confuses phishing-report collators

    120662 PhishTank reports (60% of all submissions)Reduces to just 3 260 unique domains893 users voted 550851 times on these domains, wastingusers resources that could be focused elsewhere

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

    lof80.infohttp://www.volksbank.de.netw.oid3614061.lof80.info/vr

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Outline

    1 Introduction to PhishTank data collection and analysis

    2 Testing the accuracy of PhishTanks crowd decisions

    3 Disrupting PhishTanks verification system

    4 Comparing open and closed phishing feeds

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Miscategorization in PhishTank

    Nearly all submitted URLs are verified as phishing only 3%are voted down as invalid

    Many invalid URLs are still dubious 419 scams, malwarehosts, mule-recruitment sites

    Even moderators sometimes get it wrong 1.2% of theirsubmissions are voted down

    PhishTank rewrites history when it is wrong, so we couldidentify 39 false positives and 3 false negatives

    False positives include real institutions: ebay.com, ebay.de,53.com, nationalcity.comFalse negatives include a rock-phish domain already voteddown previously

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

    ebay.comebay.de53.comnationalcity.com

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Does experience improve user accuracy?

    1

    210

    111

    00

    101

    1000

    1001

    10

    000

    1000

    110

    0000

    1000

    01 up

    Perc

    enta

    ge0

    1020

    3040

    50

    Invalid submissionsDisputed votes

    1

    210

    111

    00

    101

    1000

    1001

    10

    000

    1000

    110

    0000

    Perc

    enta

    ge0

    1020

    3040

    50Figure: Inaccuracy of user submissions and votes according to the totalnumber of submissions and votes per user, respectively (left). Proportionof all invalid user submissions grouped by number of submissions (right).

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Do users with bad voting records vote together?

    High-conflict users: HC=93 users where most votes are bad

    Empirical measure of voting overlap

    overlap(HC) =

    AHC

    BHC,B 6=A

    |VA VB | = 254

    Expected overlap if relationship between users is random(thanks to Jaeyeon Jung)

    E(overlap) =

    AHC

    BHC,B 6=A

    min(|VA|,|VB|)

    i=1

    i

    (

    |VA|i

    )

    (|T ||VA|

    |VB |i

    )

    ( |T ||VB |

    )= 0.225

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Outline

    1 Introduction to PhishTank data collection and analysis

    2 Testing the accuracy of PhishTanks crowd decisions

    3 Disrupting PhishTanks verification system

    4 Comparing open and closed phishing feeds

    Tyler Moore Evaluating the Wisdom of Crowds in Assessing Phishing Sites

  • Introduction to PhishTank data collection and analysisTesting the accuracy of PhishTanks crowd decisions

    Disrupting PhishTanks verification systemComparing open and closed phishing feeds

    Disrupting PhishTanks verification system

    Can PhishTanks open submission and voting policies beexploited by attackers?

    Other anti-phishing groups have been targeted by DDoSattacks

Recommended

View more >