crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · topics • crowdfunding...
TRANSCRIPT
![Page 1: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/1.jpg)
CrowfordCrowd Investment Data Portal
Group 13 LABBE, Kevin Patrick Joseph
MARTYNAVA, Karina THOMPSON, Julien Edward
![Page 2: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/2.jpg)
Topics
• Crowdfunding basics
• Schema Mapping / ER (Gathering Data)
• Data Fusion (Data Analysis)
• Data Portal
![Page 3: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/3.jpg)
Crowdfunding
! Crowdfunding
! Schema mapping / ER
! Data fusion
! Data Portal
![Page 4: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/4.jpg)
Crowdfunding• Fund a project by a large number of people
• Start-up, Video games, charity…
! Crowdfunding
! Schema mapping / ER
! Data fusion
! Data Portal
![Page 5: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/5.jpg)
Crowdfunding
• over 2 billions $
• 100K projects
• 10 milion contributors
! Crowdfunding
! Schema mapping / ER
! Data fusion
! Data Portal
![Page 6: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/6.jpg)
Crowford• Gather projects from different sources
• Predict if a project will be successful or not
! Crowdfunding
! Schema mapping / ER
! Data fusion
! Data Portal
![Page 7: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/7.jpg)
Schema Mapping / ER
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
Gathering data
![Page 8: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/8.jpg)
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data PortalData Sources
![Page 9: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/9.jpg)
• Same structure
• Same theme (fund projects)
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data PortalData Sources
![Page 10: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/10.jpg)
Crowdfund data
Project
Idea (Pen, Video game, Product…)
Packages / Perks What you get / Money
Author(s)
![Page 11: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/11.jpg)
![Page 12: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/12.jpg)
![Page 13: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/13.jpg)
![Page 14: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/14.jpg)
Crawling! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
![Page 15: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/15.jpg)
Crawling
• Use JavaScript to generate project page
• Private API that generate JSON
• Python script w/ http2 that generate requests
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
![Page 16: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/16.jpg)
https://www.indiegogo.com/private_api/explore?filter_funding=&filter_percent_funded=&filter_q
uick=new&filter_status=&pg_num=2
![Page 17: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/17.jpg)
Crawling
• Have to extract data from websites
• 2 Spider bots (crawlers) using Scrapy
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
![Page 18: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/18.jpg)
Crawling w/ Scrapy
• Python Framework for extracting data
• Write Spider (crawling bots)
• Parse data and extract with xpath
• Export data (Schema mapping)
![Page 19: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/19.jpg)
![Page 20: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/20.jpg)
• Initialize the spider
![Page 21: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/21.jpg)
• Download and extract data
• Export the item
![Page 22: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/22.jpg)
• Export the item
![Page 23: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/23.jpg)
Data scheme
Author
Project Perks / Packages
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
![Page 24: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/24.jpg)
Project
Author
Project Summary Perks / Packages
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data PortalData scheme
![Page 25: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/25.jpg)
Data scheme
• Project / Author
• Project / Perks
• Recommendation : Project / Related_Project
• ER for multiple authors / perks / projects…
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
![Page 26: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/26.jpg)
Results
• Our working set
• 36 000 projects
• 65 000 authors
• over 230 000 perks
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
![Page 27: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/27.jpg)
Data fusion
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
And other data analysis
![Page 28: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/28.jpg)
Recommendation• Use buzzwords in project description
• Use n-grams (word combination)
• Similarity measures using Pairwise metrics
• Linear kernels
• Can be used for data fusion
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
![Page 29: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/29.jpg)
Success Prediction
• How much money has been collected
• How much time
• The average pledge
• …
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
![Page 30: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/30.jpg)
Success PredictionRandom Forests Logistic regression
RESULT 1 RESULT 2
COMPARISON
RESULT
! Crowdfunding
! Schema mapping / ER
! Data Fusion
! Data Portal
![Page 31: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/31.jpg)
Data Portal
! Crowdfunding
! Schema mapping / ER
! Data fusion
! Data Portal
![Page 32: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/32.jpg)
Goal
Browse successful projects
! Crowdfunding
! Schema mapping
! Data fusion
! Data Portal
![Page 33: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/33.jpg)
DataBase
PostgreSQL
![Page 34: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/34.jpg)
DataBase
PostgreSQL
Web Interface
Django
![Page 35: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/35.jpg)
DataBase
PostgreSQL
Web Interface
Django
Project List Project Info
![Page 36: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/36.jpg)
Web Interface
• Project list
• Filter
• Access project page
• Allows you to download the datasets
! Crowdfunding
! Schema mapping
! Data fusion
! Data Portal
![Page 37: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/37.jpg)
Demo
![Page 38: Crowford - home.cse.ust.hkhome.cse.ust.hk/~leichen/courses/mscit6000d... · Topics • Crowdfunding basics • Schema Mapping / ER (Gathering Data) • Data Fusion (Data Analysis)](https://reader033.vdocuments.site/reader033/viewer/2022052101/603a9e50927ebc1e0c034f65/html5/thumbnails/38.jpg)