workshop: science meets industry: online behavioral experiments with nodegame - stefano balietti,...
TRANSCRIPT
![Page 1: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/1.jpg)
![Page 2: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/2.jpg)
Online Behavioral Experiments
with NodeGame
Stefano BaliettiNetwork Science Institute, Northeastern University
Harvard Institute for Quantitative Social Science
D'Amore McKim School of Business, Northeastern University
@balietti | @nodegameorg | stefanobalietti.com | nodegame.org
![Page 3: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/3.jpg)
1. Become confident about the principles and the methodology
behind social science scientific experiments
2. Get a quick-start with the software nodeGame to conduct
experimental research online
Goals of the Workshop
![Page 4: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/4.jpg)
Workshop Instructors
![Page 5: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/5.jpg)
Workshop Instructors
Me
Social Scientist Open Source
Developer
![Page 6: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/6.jpg)
Workshop Instructors
Me
Social Scientist
Open
Source
Developer
![Page 7: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/7.jpg)
What is an experiment?
![Page 8: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/8.jpg)
An experiment is a methodological procedure carried out with the
goal of verifying, falsifying, or establishing the validity of a hypothesis.
A test under controlled conditions that is made to demonstrate a
known truth, examine the validity of a hypothesis, or determine the
efficacy of something previously untried.
An experiment is an empirical method that arbitrates between
competing hypotheses.
What is an experiment?
![Page 9: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/9.jpg)
Internal validity:
The extent to which a causal inference based on a study is
warranted by an experiment.
What is an experiment?
![Page 10: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/10.jpg)
Control
3 Pillars of Experimental Design
RandomizationIndependence
![Page 11: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/11.jpg)
Control
3 Pillars of Experimental Design
RandomizationIndependence
Deliberate choice of the variables
that will be kept constants and those
that will be varied: treatment variables
![Page 12: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/12.jpg)
Control
3 Pillars of Experimental Design
Randomization
Independence
Treatment variables must not be correlated
![Page 13: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/13.jpg)
Control
3 Pillars of Experimental Design
Randomization
Independence
Participants are randomly assigned to conditions
Conditions are executed in random order
Stratification might be required
![Page 14: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/14.jpg)
Internal validity:
The extent to which a causal inference based on a study is
warranted by an experiment.
External validity:
The extent to which the results of a study can be generalized
or extended to others.
What is an experiment?
![Page 15: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/15.jpg)
Internal validity:
The extent to which a causal inference based on a study is
warranted by an experiment.
External validity:
The extent to which the results of a study can be generalized
or extended to others.
What is an experiment?
![Page 16: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/16.jpg)
External Validity
![Page 17: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/17.jpg)
Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610)
External Validity
![Page 18: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/18.jpg)
Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610)
External Validity
Telescope:
From Land to Sky
![Page 19: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/19.jpg)
Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610)
External Validity
Telescope:
From Land to Sky
Experiments:
From Lab to Real Life
![Page 20: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/20.jpg)
Super-brief history of scientific experiments
![Page 21: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/21.jpg)
• Novum Organum (1620)
• Scientific induction
• Torturing Nature to extract her secrets
Super-brief history of scientific experiments (1/3)
![Page 22: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/22.jpg)
• Novum Organum (1620)
• Scientific induction
• Torturing Nature to extract her secrets
Super-brief history of scientific experiments (1/3)
Ever heard of Phlogistons??
![Page 23: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/23.jpg)
“There is a property common to almost all
the moral sciences, and by which they are
distinguished from many of the physical;
that is, that it is seldom in our power to
make experiments with them”
John Stuart Mill, 1836
Super-brief history of scientific experiments (2/3)
![Page 24: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/24.jpg)
“There is no laboratory in which economists can test
their hypothesis”
, 1993
Super-brief history of scientific experiments (2B/3)
![Page 25: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/25.jpg)
Nobel prize for Vernon Smith
“for having established laboratory
experiments as a tool in empirical
economic analysis, especially in the
study of alternative market mechanisms”
2002
Super-brief history of scientific experiments (3/3)
![Page 26: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/26.jpg)
The Puzzle of Cooperation
![Page 27: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/27.jpg)
The Ultimatum Game
![Page 28: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/28.jpg)
The Ultimatum Game
Rules:
• 2 Players (A,B)
• 100 $
![Page 29: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/29.jpg)
The Ultimatum Game
Rules:
• 2 Players (A,B)
• 100 $
• Player A makes a proposal
![Page 30: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/30.jpg)
The Ultimatum Game
Rules:
• 2 Players (A,B)
• 100 $
• Player A makes a proposal
• Player B can accept or refuse the split.
![Page 31: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/31.jpg)
The Ultimatum Game
Rules:
• 2 Players (A,B)
• 100 $
• Player A makes a proposal
• Player B can accept or refuse the split.
• If B accepts each one gets his/her
share.
![Page 32: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/32.jpg)
The Ultimatum Game
Rules:
• 2 Players (A,B)
• 100 $
• Player A makes a proposal
• Player B can accept or refuse the split.
• If B accepts each one gets his/her
share.
• If B refuses both players get nothing.
![Page 33: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/33.jpg)
The Ultimatum Game: How to Split 100$
Rules:
• 2 Players (A,B)
• 100 $
• Player A makes a proposal
• Player B can accept or refuse the split.
• If B accepts each one gets his/her
share.
• If B refuses both players get nothing.
![Page 34: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/34.jpg)
The Ultimatum Game: How to Split 100$
A B
![Page 35: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/35.jpg)
The Ultimatum Game: How to Split 100$
Proposer RespondentRationalRational
![Page 36: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/36.jpg)
The Ultimatum Game: How to Split 100$
Proposer Respondent
I say 99 for me
and 1 for you.
RationalRational
![Page 37: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/37.jpg)
The Ultimatum Game: How to Split 100$
Proposer Respondent
I say 99 for me
and 1 for you.
RationalRational
I say OK.
![Page 38: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/38.jpg)
The Ultimatum Game: How to Split 100$
Human Being Human Being
I say 99 for me
and 1 for you.
![Page 39: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/39.jpg)
The Ultimatum Game: How to Split 100$
Human Being Human Being
I say…$!!@**I say 99 for me
and 1 for you.
![Page 40: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/40.jpg)
Nowak (2006) 5 rules for Cooperation
![Page 41: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/41.jpg)
Traditional Laboratory Research
• Decision making tools, heuristics, models of human behavior
![Page 42: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/42.jpg)
Traditional Laboratory Research
• Decision making tools, heuristics, models of human behavior
• BUT
![Page 43: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/43.jpg)
Traditional Laboratory Research
• Decision making tools, heuristics, models of human behavior
• BUT
• Small groups of individuals, generally 4-12 (bounded by uni. Lab)
![Page 44: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/44.jpg)
Traditional Laboratory Research
• Decision making tools, heuristics, models of human behavior
• BUT
• Small groups of individuals, generally 4-12 (bounded by uni. Lab)
• Behavior of individuals in social networks under-researched
![Page 45: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/45.jpg)
Traditional Laboratory Research
• Decision making tools, heuristics, models of human behavior
• BUT
• Small groups of individuals, generally 4-12 (bounded by uni. Lab)
• Behavior of individuals in social networks under-researched
• Limited in the number of hypotheses that can be tested
![Page 46: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/46.jpg)
Traditional Laboratory Research
• Decision making tools, heuristics, models of human behavior
• BUT
• Small groups of individuals, generally 4-12 (bounded by uni. Lab)
• Behavior of individuals in social networks under-researched
• Limited in the number of hypotheses that can be tested
• Observations are limited in time
![Page 47: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/47.jpg)
Traditional Laboratory Research
• Decision making tools, heuristics, models of human behavior
• BUT
• Small groups of individuals, generally 4-12 (bounded by uni. Lab)
• Behavior of individuals in social networks under-researched
• Limited in the number of hypotheses that can be tested
• Observations are limited in time
• Limited(?) external validity
![Page 48: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/48.jpg)
Traditional Laboratory Research
• Decision making tools, heuristics, models of human behavior
• BUT
• Small groups of individuals, generally 4-12 (bounded by uni. Lab)
• Behavior of individuals in social networks under-researched
• Limited in the number of hypotheses that can be tested
• Observations are limited in time
• Limited(?) external validity
• Generally high costs for an academic budget
![Page 49: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/49.jpg)
The Present/Future of Behavioral Research
![Page 50: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/50.jpg)
The Present/Future of Behavioral Research
![Page 51: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/51.jpg)
Workshop Instructors
Me
Social Scientist Open Source
Developer
![Page 52: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/52.jpg)
Workshop Instructors
Me
Social Scientist
Open Source Developer
![Page 53: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/53.jpg)
nodeGame 3.x
![Page 54: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/54.jpg)
Install nodeGame 3.x
1. Go to http://nodegame.org
2. Click on the “Get Started” button
3. Follow instructions to install Development version
4. Keep following my talk!
![Page 55: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/55.jpg)
nodeGame 3.x Architecture Overview
![Page 56: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/56.jpg)
nodeGame 3.x Architecture Overview
![Page 57: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/57.jpg)
nodeGame 3.x Architecture Overview
![Page 58: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/58.jpg)
nodeGame 3.x Architecture Overview
~46.000 lines
(including comments)
![Page 59: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/59.jpg)
Launch nodeGame 3.x
Open a terminal and browse to the nodegame folder
Start the server with the command: node launcher.js
Open a browser and access the default ultimatum game at the url:localhost:8080/ultimatum
Start an automated player (bot) at the url:localhost:8080/ultimatum?clientType=autoplay
Access the administrator interface at the url:localhost:8080/ultimatum/monitor/
![Page 60: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/60.jpg)
Create a New Game
cd nodegame/bin/
node nodegame create-game gamename <author> <email>
Creates a new game in the games/ folder (usually nodegame/games/)
Important! Windows users must install the nodegame-generator package
manually from the git repository (or npm)
git clone https://github.com/nodeGame/nodegame-generator.git
cd nodegame-generator
npm install
![Page 61: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/61.jpg)
Anatomy of a Game
![Page 62: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/62.jpg)
Anatomy of a Game
Features:- Define game variables and treatments
- Define the game sequence with the Stager API
- Implement the game sequence into client types
![Page 63: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/63.jpg)
The Game Sequence and Client Types
Game Sequence Client Typesstager.stage("instructions")
.step("instructions_1")
.step("instructions_2")
.step("instructions_3");
stager.stage("quiz");
stager.repeat("game", 3)
stager.stage("questionnaire");
stager.extendStage("game", {
steps: ["offer", "respond",
. "display_results"]
});
Code Snippet
![Page 64: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/64.jpg)
The Game Sequence and Client Types
Game Sequence Client Types
stager.stage("instructions")
.step("instructions_1")
.step("instructions_2")
.step("instructions_3");
stager.stage("quiz");
stager.repeat("game", 3)
stager.stage("questionnaire");
stager.extendStage("game", {
steps: [ "offer", "respond",
. "display_results" ]
});
Code Snippet
![Page 65: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/65.jpg)
The Game Sequence and Client Types
Game Sequence Client Types
Client types implement the game sequence
by extending its steps and stages.
![Page 66: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/66.jpg)
The Game Sequence and Client Types
Game Sequence Client Types
![Page 67: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/67.jpg)
The Game Sequence and Client Types
Game Sequence Client Types
![Page 68: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/68.jpg)
The Game Sequence and Client Types
Game Sequence Client Types Code Snippet
stager.setDefaultProperty("coins",100);
stager.extendStage("game", {
coins: 200
});
stager.extendStep("offer", {
coins: 50
});
![Page 69: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/69.jpg)
The Game Sequence and Client Types
Game Sequence Client Types Code Snippet
stager.setDefaultProperty("coins",100);
stager.extendStage("game", {
coins: 200
});
stager.extendStep("offer", {
coins: 50
});
node.game.getProperty("coins") returns 50 in
step 'offer', 200 in steps 'respond' and 'display results'
inside the 'game' stage, and 100 in any other step.
Step/Stages Properties are hierarchical
![Page 70: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/70.jpg)
Some Built-in Step-Properties
Property Description Type
init Executed upon entering the step/stage function
exit Executed upon exiting the step/stage function
timer Sets the maximum execution time for step (ms) number
timeup Executed if the timer expires function
done Executed when the step terminates function
cb The actual step execution function
![Page 71: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/71.jpg)
Logic Client Type
Executed on the server, governs a game room
- Makes groups
- Keep track of game history
- Handles disconnections and reconnections
- Access to:- internal Database NDDB: node.game.memory
- player list: node.game.pl
- channel and registry object
![Page 72: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/72.jpg)
Player Client Types
Executed on a client, e.g. a browser
- Display/manipulate information on screen
- Exchange messages with the server and other players
- Respond to user events
- Determine when the current step is finished (“done”)
W (Window): manipulates information on screen and the behavior of user interface,
e.g. disabling right click, locking screen, etc.
At the end of each step, the game developer must call the methodnode.done, which triggers a procedure that advances the client to the next step.
![Page 73: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/73.jpg)
Inside a Step Callback Function
Instructions Step
- Player client type
- No interaction with other players
- No interaction with the page
- No time limit
- Simply click “Done” when
finished reading and advance to
next step
![Page 74: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/74.jpg)
Inside a Step Callback Function
Instructions Step
- Player client type
- No interaction with other players
- No interaction with the page
- No time limit
- Simply click “Done” when
finished reading and advance to
next step
![Page 75: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/75.jpg)
Inside a Step Callback Function
stager.extendStep('instructions_1', {
frame: "instr1.htm"
});
Instructions Step
- Player client type
- No interaction with other players
- No interaction with the page
- No time limit
- Simply click “Done” when
finished reading and advance to
next step
![Page 76: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/76.jpg)
Inside a Step Callback Function
stager.extendStep('instructions_1', {
frame: "instr1.htm"
});
Widget Step
- Player client type
- No interaction with other players (at
the moment)
- Standardized interaction
- With or without time limit
- Register the mood of the participant
using the PANAS method.
Instructions Step
- Player client type
- No interaction with other players
- No interaction with the page
- No time limit
- Simply click “Done” when
finished reading and advance to
next step
![Page 77: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/77.jpg)
Inside a Step Callback Function
stager.extendStep('instructions_1', {
frame: "instr1.htm"
});
Widget Step
- Player client type
- No interaction with other players (at
the moment)
- Standardized interaction
- With or without time limit
- Register the mood of the participant
using the PANAS method.
Instructions Step
- Player client type
- No interaction with other players
- No interaction with the page
- No time limit
- Simply click “Done” when
finished reading and advance to
next step
![Page 78: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/78.jpg)
Inside a Step Callback Function
stager.extendStep('instructions_1', {
frame: "instr1.htm"
});
Widget Step
- Player client type
- No interaction with other players (at
the moment)
- Standardized interaction
- With or without time limit
- Register the mood of the participant
using the PANAS method.
Instructions Step
- Player client type
- No interaction with other players
- No interaction with the page
- No time limit
- Simply click “Done” when
finished reading and advance to
next step
stager.extendStep('instructions_2', {
widget: "MoodGauge"
});
![Page 79: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/79.jpg)
Roles (version 3.5 dev)
stager.extendStep('bidder', {
matcher: {
// Available roles.
roles: [ 'BIDDER', 'RESPONDENT'],
// Each player is matched with a
// partner and is assigned a role.
match: 'roundrobin',
// How to continue matching after
// all combinations are exhausted
cycle: 'repeat_invert'
}
});
Logic Client Type
![Page 80: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/80.jpg)
Roles (version 3.5 dev)
stager.extendStep('bidder', {
roles: {
BIDDER: {
frame: 'bidder.htm',
cb: function() {
// Bidder stuff.
}
},
RESPONDENT: {
frame: 'resp.htm',
cb: function() {
// Respondent stuff.
}}}
});
Player Client Type
stager.extendStep('bidder', {
matcher: {
// Available roles.
roles: [ 'BIDDER', 'RESPONDENT'],
// Each player is matched with a
// partner and is assigned a role.
match: 'roundrobin',
// How to continue matching after
// all combinations are exhausted
cycle: 'repeat_invert'
}
});
Logic Client Type
![Page 81: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/81.jpg)
Inside the Step Callback Function
// The respondent receives the offer,
// displays it, and terminates the step.
// On incoming data labeled "offer",
// executes the function.
node.on.data("offer", function(msg) {
// Display offer using W.
W.setInnerHTML("offer", msg.data);
// End step.
node.done();
});
// Player made a choice
// and clicked the submit button.
submit.onclick = function() {
// Send offer to respondent.
node.say("offer", "respondent", 1);
// End step, store offer in server.
node.done({ offer: 1 });
};
I say 99 for me
and 1 for you.
Proposer Respondent
![Page 82: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/82.jpg)
NDDB: Internal Database
// The respondent receives the offer,
// displays it, and terminates the step.
// On incoming data labeled "offer",
// executes the function.
node.on.data("offer", function(msg) {
// Display offer using W.
W.setInnerHTML("offer", msg.data);
// End step.
node.done();
});
// Player made a choice
// and clicked the submit button.
submit.onclick = function() {
// Send offer to respondent.
node.say("offer", "respondent", 1);
// End step, store offer in server.
node.done({ offer: 1});
};
I say 99 for me
and 1 for you.
Proposer RespondentLogic Client Type
NDDB database
node.game.memory
![Page 83: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/83.jpg)
NDDB: Internal Database
node.done({ offer: 1 });
// Set an object in memory explicitly.
node.set({ foo: "bar" });
![Page 84: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/84.jpg)
NDDB: Internal Database
{
stage: {
stage: 2,
step: 1,
round: 1
},
player: "respondent",
timestamp: 1480336721939
offer: 1, // foo: "bar"
// node.done specific.
timeup: false,
time: 4321, // from step.
done: true
};
node.done({ offer: 1 });
// Set an object in memory explicitly.
node.set({ foo: "bar" });
![Page 85: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/85.jpg)
NDDB: Internal Database
var m = node.game.memory; // NDDB instance.
// Lazy loading.
m.select("player", "=", "respondent");
m.and("stage.step", "=", 1)
.and("offer")
.fetch()[0];
// Use indexes.
m.player["respondent"].last();
m.stage["2.1.1"].select("offer").fetch()[0];
node.done({ offer: 1 });
// Set an object in memory explicitly.
node.set({ foo: "bar" });
{
stage: {
stage: 2,
step: 1,
round: 1
},
player: "respondent",
timestamp: 1480336721939
offer: 1, // foo: "bar"
// node.done specific.
timeup: false,
time: 4321, // from step.
done: true
};
![Page 86: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/86.jpg)
NDDB: Internal Database
// Modify database items upon inserting them.
node.game.memory.on("insert", function(item) {
// Decorate item or save it somewhere else.
});
// Disable internal database completely.
// Remove default event listener.
node.off('in.set.DATA');
// Add new one.
node.on('in.set.DATA', function(msg) {
// Save msg somewhere else, e.g. Mongo.
});
Event Listener
Manager
node.game.events
![Page 87: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/87.jpg)
NDDB: Internal Database
// Modify database items upon inserting them.
node.game.memory.on("insert", function(item) {
// Decorate item or save it somewhere else.
});
// Disable internal database completely.
// Remove default event listener.
node.off('in.set.DATA');
// Add new one.
node.on('in.set.DATA', function(msg) {
// Save msg somewhere else, e.g. Mongo.
});
Event Listener
Manager
node.game.events
![Page 88: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/88.jpg)
Events in nodeGame 3.x
Internal: generated by the game or by the nodeGame engine.
Incoming: triggered by incoming messages, 'in.[target].[action]', ‘in.set.DATA’
![Page 89: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/89.jpg)
Events in nodeGame 3.x
Internal: generated by the game or by the nodeGame engine.
Incoming: triggered by incoming messages, 'in.[target].[action]', ‘in.set.DATA’
// Registering an internal event listener.
node.on('LOCAL_EVENT', function(a, b) {
// This is local event.
});
// Triggering a local event.
node.emit('LOCAL_EVENT', 1, 2);
![Page 90: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/90.jpg)
Events in nodeGame 3.x
Internal: generated by the game or by the nodeGame engine.
Incoming: triggered by incoming messages, 'in.[target].[action]', ‘in.set.DATA’
// Registering an internal event listener.
node.on('LOCAL_EVENT', function(a, b) {
// This is local event.
});
// Listening for incoming DATA msgs.
node.on('in.say.DATA', function(msg) {
// I got a data msg!
});
// Listening for incoming "foo" DATA msgs.
node.on.data('foo', function(msg) {
// I got a data msg labeled "foo"!
});
// Triggering a local event.
node.emit('LOCAL_EVENT', 1, 2);
// Triggering a local event.
node.say('foo', "bar");
![Page 91: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/91.jpg)
Events in nodeGame 3.x
Events are hierarchically organized, where you register your event matters!
![Page 92: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/92.jpg)
Events in nodeGame 3.x
Events are hierarchically organized, where you register your event matters!
// Listening for incoming "foo" DATA msgs.
node.on.data('foo', function(msg) {
// I got a data msg labeled "foo"!
});
![Page 93: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/93.jpg)
Events in nodeGame 3.x
Events are hierarchically organized, where you register your event matters!
// Listening for incoming "foo" DATA msgs.
node.on.data('foo', function(msg) {
// I got a data msg labeled "foo"!
});
Valid only in Quiz stage
![Page 94: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/94.jpg)
Events in nodeGame 3.x
Events are hierarchically organized, where you register your event matters!
// Listening for incoming "foo" DATA msgs.
node.on.data('foo', function(msg) {
// I got a data msg labeled "foo"!
});
Registered in init or cb function of Step 3.3
Valid only in step 3.3 (any round).
![Page 95: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/95.jpg)
Events in nodeGame 3.x
Events are hierarchically organized, where you register your event matters!
// Listening for incoming "foo" DATA msgs.
node.on.data('foo', function(msg) {
// I got a data msg labeled "foo"!
});
Registered in the init function of “Game”
Valid throughout all steps of “Game” stage
![Page 96: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/96.jpg)
Events in nodeGame 3.x
Events are hierarchically organized, where you register your event matters!
How to register a listener valid throughout all
the game?
![Page 97: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/97.jpg)
Events in nodeGame 3.x
Events are hierarchically organized, where you register your event matters!
stager.setInitCb(function() {
// Listening for incoming "foo" DATA msgs.
node.on.data('foo', function(msg) {
// I got a data msg labeled "foo"!
});
});
How to register a listener valid throughout all
the game?
In the init function of the game!
![Page 98: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/98.jpg)
Anatomy of a Game
![Page 99: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/99.jpg)
Topics Covered and To Cover
![Page 100: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/100.jpg)
Topics Covered and To Cover
![Page 101: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/101.jpg)
Recruiting Participants
Free Platforms
Psychological Research on the Net http://psych.hanover.edu/research/exponnet.html
The Web Experiment List http://www.wexlist.net
Online Social Psychology Studieshttp://www.socialpsychology.org/expts.htm
Volunteer Sciencehttp://www.volunteerscience.com
![Page 102: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/102.jpg)
Recruiting Participants
Commercial Platforms
ODesk, Freelancer, Elance, Guru, CrowdFlower, Innocentive, SurveySampling,
UpWork …
Most used in research:
![Page 103: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/103.jpg)
How To Run a Game on AMT:
The Big Picture
1. Upload nodeGame server + game to a servere.g. Heroku cloud: https://github.com/nodeGame/nodegame-heroku
2. Create a new HIT on Amazon Mechanical Turk (AMT), e.g. using the requester
interface or other service API.
3. Use the nodegame-mturk package https://github.com/nodeGame/nodegame-mturk/
to handle:
- Connection and authentication from AMT HIT page to nodeGame server.
- Opening/expiring/extending HIT while running it.
- Payments, bonuses and qualifications after HIT is done.
![Page 104: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/104.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
![Page 105: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/105.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
![Page 106: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/106.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
![Page 107: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/107.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
4. Demographics should be stratified as much as possible
![Page 108: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/108.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
4. Demographics should be stratified as much as possible
5. Participants should not re-take the same experiment again
![Page 109: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/109.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
4. Demographics should be stratified as much as possible
5. Participants should not re-take the same experiment again
6. Dropouts rate (non-completing the HIT) signigicantly affects the
whole experiment, increasing costs $$ and time to complete it
![Page 110: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/110.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
Pay minimum wage (at least). Make bonus dependent on
performance
![Page 111: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/111.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
Rule of thumb: 2.8 times desidered number of participants
Expire HIT as soon as desired number of participant entered
the experimental pipeline (+small buffer)
3x
![Page 112: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/112.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
Rule of thumb: 2.8 times desidered number of participants
Expire HIT as soon as desired number of participant entered
the experimental pipeline (+small buffer)
Dispose participants who waited too long (compensated)
![Page 113: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/113.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
4. Demographics should be stratified as much as possible
Waiting room can group players based on features
Collect some demographics features in a pre-experimental part
Start 2 (or more) groups at the same time
![Page 114: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/114.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
5. Participants should not re-take the same experiment again
Create one HIT (no group) with 3x rule
Keep track of turker ids (nodegame-mturk)
Assign qualification via AMT API (nodegame-mturk)
![Page 115: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/115.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
6. Rate of dropouts (non-completing the HIT) significantly affects the
whole experiment, increasing costs $$ and time to complete it
• Make sure waiting time is not too long (previous advices)
![Page 116: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/116.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
6. Rate of dropouts (non-completing the HIT) significantly affects the
whole experiment, increasing costs $$ and time to complete it
• Make sure waiting time is not too long (previous advices)
• Inform about required time BEFORE and DURING experiment
![Page 117: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/117.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
6. Rate of dropouts (non-completing the HIT) significantly affects the
whole experiment, increasing costs $$ and time to complete it
• Make sure waiting time is not too long (previous advices)
• Inform about required time BEFORE and DURING experiment
• Design experiments that are robust to dropout (as much as
possible). For example, use shares instead of fixed quantities.
![Page 118: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/118.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
6. Rate of dropouts (non-completing the HIT) significantly affects the
whole experiment, increasing costs $$ and time to complete it
• Make sure waiting time is not too long (previous advices)
• Inform about required time BEFORE and DURING experiment
• Design experiments that are robust to dropout (as much as
possible). For example, use shares instead of fixed quantities.
• Use requirements checkings.
![Page 119: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/119.jpg)
Known Issues Conducting
Synchronous Experiments on AMT
6. Rate of dropouts (non-completing the HIT) significantly affects the
whole experiment, increasing costs $$ and time to complete it
• Make sure waiting time is not too long (previous advices)
• Inform about required time BEFORE and DURING experiment
• Design experiments that are robust to dropout (as much as
possible). For example, use shares instead of fixed quantities.
• Use requirements checkings.
• Use “seriousness checks” and “warm-up” phases.
![Page 120: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/120.jpg)
How To Run a Game on AMT
![Page 121: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/121.jpg)
AMT
How To Run a Game on AMT
AMT HIT
![Page 122: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/122.jpg)
Not Just Experiments,
also Gamification
• Industry: lots of A/B testing and customer surveys
• Gamification: applies elements of game playing to solve
complex and simple problems
• Increasing trend
• Improve engagement and understanding
• Fun
• Generate More and Better Ideas
• Train Machine Learning Algorithms
(http://quickdraw.withgoogle.com)
![Page 123: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/123.jpg)
Contributions Welcome!
• Using it, request features, report bugs
• Help developing
• Extremely well documented source code
![Page 124: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/124.jpg)
Contributions Welcome!
• Using it, request features, report bugs
• Help developing
• Extremely well documented source code
![Page 125: Workshop: Science Meets Industry: Online Behavioral Experiments with nodeGame - Stefano Balietti, Northeastern University](https://reader031.vdocuments.site/reader031/viewer/2022030301/587fa9291a28ab825e8b794b/html5/thumbnails/125.jpg)
Thank You!
@balietti | @nodegameorg | stefanobalietti.com | nodegame.org