2019 - infommmi - intro · week 8 analysis & report / paper writing week 9 project presentation...

Multimodal Interaction, Feb 6, 2019

IntroductionWolfgang Hürst

Contact: [email protected], http://people.cs.uu.nl/huerst

1

Contact information

Lectures 1-4, guest lecture & examPeter Werkhoven, UU & [email protected]

Lectures 5-8, exam & projectWolfgang Hürst, UU, Multimedia [email protected], BBG 480(contact me also for administrational questions)

When you email us, start subject with [INFOMMMI]

2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 2

mailto:[email protected]

mailto:[email protected]

IMPORTANT!

Make sure to regularly check your student email, since some important information, last minute

changes, etc. might be sent by email.

It is also a good idea to keep an eye on the “news” page of the course’s website.


Multimodal interaction

General Introduction


How are we interacting with computers?


KEYBOARD, MOUSE

GRAPHICAL USER INTERFACES

(MULTI) TOUCH

SPEECH

GESTURES & MOTION

DIFFERENT DISPLAYS

Computer usage – from data to experience


1D 3D

Punch cards & cables, …

data information & communication

experience

Touch (& mouse) gestures, …

CLI (Command Line Interface)

Keyboard & mouse

Trackpads, touchscreens, …

Body motions / gestures, tangible UIs, …

Speech, …

Mainframes & workstations PCs & internet Mobile devices,

HMDs, projective displays, …

GUI (Graphical User Interface)

From to to

2D


This requires• Looking into different technologies• Looking into human aspects

(e.g. perception of visual, tactile, and acoustic feedback)

What modalities make sense?And when? And why?

Goal of this course: Learn about different modalities for interaction and when and how to use them.

Overview(lectures)

Human perception• Hearing• Vision• Touch• Multi-sensory• Synaesthesia

Virtual reality• Applications• Technology• Interaction• Navigation• Mobile VR

Augmented reality• Technology• Human factors• Interaction

Lectures are complemented with papers for mandatory reading.Make sure to read them in time, so you have enough time for the project.

Example for human perception …

2/6/19 9W. Hürst, http://people.cs.uu.nl/huerst

Source: National Geographic – Brain Gameshttp://natgeotv.com/nl/brain-games/videos/rubberen-hand

http://natgeotv.com/nl/brain-games/videos/rubberen-hand

Integration of visual and tactile stimuli in a stereoscopic virtual environment

Goal: investigate perception of incongruencies in location between tactile and visual stimuli

Lecture Project Recommended actions

Week 1Introduction

Papers 1, 5

Week 2 Lecture 1 (vision)Lecture 2 (sound, …) Deadline for forming groups

Papers 2, 3, 4 & forming project groups

Week 3 Lecture 3 (VR)Lecture 4 (other)

Papers 6, 7, 8 & work on project ideas, prepare pitches, plan

Week 4 Project pitchesGuest lecture

Deadline pitch slides: Feb 25, 9:00Deadline project plan: Feb 27, 23:59

Pitch ideas & make project plan

Week 5 Lecture 5 (AR)Lecture 6 (AR)

AR papers &setup and implementation

Week 6 Project presentationLecture 7 (AR)

Deadline slides: Mar 11, 9:00 AR papers &setup and implementation

Week 7 Lecture 8 (AR) Experiments

Week 8 Analysis & report / paper writing

Week 9Project presentation(final) & meetings

Deadline project report: Apr 1, 23:59

Examweek

Exam

GradingGRADING PASSING

EXAM (lectures 1-4, guest, lectures 5-8)

Exam grade counts 50% for final grade

Min. 5.0 (before rounding)

PROJECT Project grade counts 50% for final grade

Min. 5.0(before rounding)Min. 5.5 (before rounding)


Participation in all grading-related events (exam, project presentations & meetings) is mandatory.

• Participation in all events (exams, project presentations and meetings)

• Final grade at least 4.0 (after rounding)

Retake: requirements for qualification

Passing

Note: Attendance of the lectures is mandatory, too!

However, we are not actively checking it (but expect exam questions that are not covered by the slides and literature).

Attendance in the meetings and presentations related to the project is mandatory and will be checked.


Exceptions may be made if you have a time conflict (but contact me before the event).

• If you failed because your exam average or your overall grade was too low, you can do the retake exam


• If you failed because your grade in the project was too low, you can get a new assignment

Keep in mind that the retake exam is after the 4th quarter (consider this when planning your summer vacation)

Retake – Requirements for passing

• Your retake grade (exam and/or project) will replace the corresponding grade (exam and/or project).

• You pass this course if your new grade fulfills the original requirements


Multimodal interaction

Project IntroductionGeneral remarks

Why a project?

• Hands-on experience is important (esp. when dealing with interaction)

• Many of the learning goals of this projectare motivated by lacks of skills that I identified in small and thesis projects

• Mostly self-dependent work, esp. with respect to

• Coming up with research ideas & solutions• Specifying your research goals & questions

• Time planning & scheduling


Why a group project?

• Notice the difference between “working self-dependent” and “working alone”

• Lack of self-dependence is also about bringing in own ideas, discuss and defend them, dealing with rather vague specifications

• Important skills for team work, which will likely be part of your job (and your thesis project, which usually contributes to a bigger research effort)


How to build the groups?

• Size: 4-5 members should be ideal (but 3-6 may be okay, too, after discussion)

• You are free to chose your team mates (assignment by me only if necessary)

• Keep in mind your skills and experience when building your teams and specifying your goals (e.g., time is too short to learn a new programming language from scratch)


Use the spreadsheet at http://tinyurl.com/infommmi2019to sign up for a group and find team members.

http://tinyurl.com/infommmi2018

Project

Topic & taskWhat you can/have to do



In the lectures, you will learn a lot about different modalities and how we perceive them.

And how we can use technology to create multimodal interaction experiences or how multimodality affects interaction.

Based on this, you should come up with your own idea of a small multimodal interaction research project.


In general, your project will have three phases / parts:1. Setup / implementation2. Experiment3. Analysis

Notice that actual development is preferable but not essential.

Topic & tasks for project

Restrictions:• You are not allowed to hit people with a hammer• No humans or animals should be harmed• No laws should be broken

Any other multimodal experiment should be fine J

Examples for multimodal / multisensory experiments


Localization (small project)

Tactile type (MSc thesis)


Parinya Punpongsanon, Daisuke Iwai, Kosuke Sato. SoftAR: Visually Manipulating Haptic Softness Perception in Spatial Augmented Reality. IEEE Transactions on Visualization and Computer Graphics (Proceedings of IEEE International Symposium on Mixed and Augmented Reality 2015), Vol. 21, No. 11, PP. 1279-1288.

Examples for multimodal/multisensory experiments


Sungjae Hwang, John Song, and Junghyeon Gim. 2015. Harmonious Haptics: Enhanced Tactile Feedback Using a Mobile and a Wearable Device. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '15).




If you have a smartphone & smartwatch (with vibration motors), you could do a similar project

Interesting questions to study:• Different vibration patterns (see paper)• Evaluate the influence of location

(wrist vs. tablet vs. both)

If you use Google Cardboard (and a 2nd

smartphone), you could even do this in VR

You could also use a smartphone for handheld AR (and test, e.g., different feedback such as visuals vs. sound vs. tactile)



Other simple setups include• Visuals & tactile

(e.g., can you influence tactile perception with visuals?)

• Visuals & sound (also on PCs)

Note:• You don’t have to use mobiles,

but can use any tools you like.• You don’t have to do VR/AR,

but anything involving multimodality is ok.• You don’t have to use multiple modalities,

but a test with one modality is also fine, if it fits to the course’s topic

The major criteria is that you demonstrate that you learned something (useful) from the course.


In previous years, one group investigated the impact of airflow on a roller coaster VR experience.

Notice that this is an example for an experiment that did not require any actual development.


One group tested the influence of audio (e.g., scary vs. happy background music) in a simple VR environment (floating cubes).

Notice that this example required some rather simple development (i.e., a well-suited topic for beginners)


One team used vibration motors hooked up to an Arduino to study frictional effects in VR and their impact on presence


And one group put various sensors (distance measures, tactile and auditory feedback) into a head to provide navigation aid for visually impaired.

Evaluation in a setup maze environment with blindfolded participants and combinations of none, auditory, and tactile feedback.

Usage of tools (hardware & software)

You are welcome to use any hardware you have available (but make sure to guarantee availability during the whole project)

You are welcome to use existing software (but make sure to specify it in your project report)

We can provide you with: • HTC Vive VR HMDs (max. 3 are available)• Two old Oculus Rift VR HMDs (1 DK1 and 1 DK2)• Meta 1 and Meta 2 AR HMDs

Note that you might need to share them with some MSc students and there’s a lack of computers and rooms (sorry for that).

You are encouraged to take advantage of this, but not required (e.g., non-VR projects are fine, too).

We will discuss hardware assignments and availability after your pitches.



Week 1Introduction

Papers 1, 5

Week 2 Lecture 1 (vision)Lecture 2 (sound, haptics) Deadline for forming groups





Deadline pitch slides: Feb 25, 9:00Deadline proj. plan: Feb 27, 23:59









Deadline proj. report: Apr 1, 23:59

Examweek

Exam

Complaint in a previous year:


“This is way too little time for a project like this!”




• The course is 7.5 ECTS, which corresponds to 210 hours.

• The project grade counts 50%, so 105 hours.

• Multiply this by number of team members(and subtract some hours for communication overhead).

With good planning, this should be enough to do something great!




• You decide what you do.• It is part of the learning goals to:

– Find an interesting & relevant topic– Find a feasible topic!– Make a good time planning!– And stick to it!

Good planning & topic selection

• It is okay when your study has limitations …… if you can explain them and are aware of the consequences.

• It is okay to (partly) fail because you aimed too high …… if you can explain why and still achieved a learning goal

(and your original goal was realistic at the time)

• Important issues are:– That you demonstrated that you learned something (useful)

from the course– That your plans and study make sense as a whole (even if they

have limitations)

Note: you are allowed to take risks, be creative, go crazy, surprise me with cool ideas!


You can approach this from two ways:• Study a cognitive issue

(but don’t forget to address the practical impact)• Look at an applied/computer engineering problem

(but make sure to address a scientific problem)


But how to get from an ideato a research questionto scientific results?

Project

How to come up with a good research topic?Some general tips & comments


The development of a research idea

Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke Johnson, Larry Christensen, Fig. 3.1, page 64

Research problemThe issue or problem within a broad topic area

Research purposeA statement of the intent or objective of the study

HypothesisA prediction or best guess of the relation that exists among the variables

being investigated

Research questionIn quantitative research it is an interrogative sentence that asks a question

about the relation between two or more variables. In qualitative research, it is an interrogative sentence that asks a question about some process, issue,

or phenomenon to be explored.

Research topicThe broad subject matter area to be investigated

For self study


Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke Johnson, Larry Christensen, Fig. 3.1, page 64



HypothesisA prediction or best guess of the relation that exists among the variables

being investigated

Research questionIn quantitative research it is an interrogative sentence that asks a question

about the relation between two or more variables. In qualitative research, it is an interrogative sentence that asks a question about some process, issue,

or phenomenon to be explored.

Research topicThe broad subject matter area to be investigated

Notice that this is from “Educational research” (but it applies to all kinds of empirical and quantitative research)


Source: Educational Research: Quantitative, Qualitative, and Mixed Approaches by Burke

Johnson, Larry Christensen, Fig. 3.1, page 64

Research problem

Research purpose

Hypothesis

Research question

Research topic Notice the differences:• From the general context / bigger aim• To a concrete research (sub) problem• To a very concrete research questions

that can be answered with scientific methods

Example:• MSc thesis

Nina Rosa (published at ICMI’15)




Research problem

Research purpose

Hypothesis

Research question

Research topicResearch topic:The broad subject matter area to be investigated

Example:We know from literature that incongruent stimuli can change perception. Can we take advantage of such effects to create richer VR experiences by providing incongruent tactile and visual stimuli?




Research problem

Research purpose

Hypothesis

Research question

Research topic Research problemThe issue or problem within a broad topic area

Example:How does perception of tactile feedback change when incongruent visual stimuli are presented simultaneously?In particular, can we use certain visual signals to create a certain tactile perception?




Research problem

Research purpose

Hypothesis

Research question

Research topicResearch purposeA statement of the intent or objective of the study

Example:Investigate if visual stimuli that represent certain characteristics or experiences can be used to modify our perception of certain tactile stimuli accordingly.




Research problem

Research purpose

Hypothesis

Research question

Research topic Research questionIn quantitative research it is an interrogative sentence that asks a question about the relation between two or more variables. In qualitative research, it is an interrogative sentence that asks a question about some process, issue, or phenomenon to be explored.

Example:Is it possible to create the illusion of experiencingdifferent intensities of a certain property (weight or temperature) using a rather simple and unrelated type of touch (vibrations) together with compelling, type-related visuals (speed/size and color, respectively)?




Research problem

Research purpose

Hypothesis

Research question

Research topic Terminology:• Independent variables = the input or

cause (i.e., what we control)• Dependent variables = the output or

outcome whose variation is being studied (i.e., what we measure)

The purpose of an experiment is generally to verify how the latter depend on the former.Ideally, they are reflected in your research question.

Example:• Independent variables:

• Color (red, gray, blue)• Falling speed and size

• Dependent variable:• Experienced intensity of properties




Research problem

Research purpose

Hypothesis

Research question

Research topicHypothesisA prediction or best guess of the relation that exists among the variables being investigated

Example:• We expect that visuals commonly

associated with weight (speed/size) lead to an increased perception of tactile intensity

• We expect that visuals commonly associated with temperature (color) lead to an increased perception of tactile intensity

But what if we can’t make such an educated guess?




Research problem

Research purpose

Hypothesis

Research question

Research topicHypothesisA prediction or best guess of the relation that exists among the variables being investigated

Terminology• Null-hypothesis: assumes no

relationship or significant difference between groups on one variable

• Directional hypothesis: predicts an outcome in one direction based on prior literature or other reasoning

• Nondirectional hypothesis: similar to directional hypothesis, but the exact form of difference (e.g., higher, lower, more, less) is not specified




Research problem

Research purpose

Hypothesis

Research question

Research topic

Be careful! There is no general agreement on the terms on the left!

People use the same term in different ways.

But the basic procedure (from “high level/big picture” to “concretely verifiable statement”) is generally the same.




Research problem

Research purpose

Hypothesis

Research question

Research topic Global aim / general goal

Related research problem(s) that need to be solved to achieve this

(big) goal

Related research questions that need to be answered to solve these

problems

Concrete objectives that need to be addressed to answer them

This is the terminology that I normally use.


Global aim / general goal

Related research problem(s) that need to be solved to achieve this

(big) goal

Related research questions that need to be answered to solve these

problems

Concrete objectives that need to be addressed to answer them

This is the terminology that I normally use.

And don’t forget that the sub-problem

that you address has to

be relevant!

For your project

2/6/19 53

What (general) problem are you approaching?What concrete sub-problem will you address?How are you going to verify your ideas/solutions?

Three slides (strict; plus one intro & one closing).

1. Problem (general goal and concrete sub problem)

2. Solution (your idea to address the sub problem and/or the research question you want to answer, related hypothesis, and objectives)

3. Methodology (your planned approach to verify your results or proof that your solution works)

Time limit is 5 minutes (& will be strictly enforced).

Recommendation for brainstorming: Try answering these questions

Project pitch (Feb 25):

Project pitches1. Problem2. Solution3. Methodology


Note: your problem and solution specification depend on your goal.

Most of you will be doing some sort of an empirical evaluation.

It is okay though to address a more engineering problem or an HCI-related issue.

While they use different approaches, they usually follow the same ”top-down” structure illustrated before.

And they also rely on objective evaluations (which generally means your problem solution should be measurable and results should be repeatable).

Project

Scientific researchSome terminology and info


Note: also pay close attention to the methodology used in the

papers for mandatory reading and learn from them.

For self study

Using SMART criteria for setting objectives

• Specific – target a specific area for improvement.

• Measurable – quantify or at least suggest an indicator of progress.

• Assignable – specify who will do it.

• Realistic – state what results can realistically be achieved, given available resources.

• Time-related – specify when the result(s) can be achieved.

Generally applied to project management, employee-performance management and personal development, but used (for example) in research proposals as well.

Helpful for your course project (and later small project & master thesis)


See, e.g., https://en.wikipedia.org/wiki/SMART_criteria

You

See planning & limitations discussed before

These are also characteristic for research projects!

https://en.wikipedia.org/wiki/SMART_criteria


Using SMART criteria for setting objectives

• Specific – target a specific area for improvement.

• Measurable – quantify or at least suggest an indicator of progress.

• Assignable – specify who will do it.

• Realistic – state what results can realistically be achieved, given available resources.

• Time-related – specify when the result(s) can be achieved.

Be careful: the term “objectives” is used in different contexts, e.g.:• High level / general goal, e.g.,

Create a SW Kit for a tactile VR suit• Specific / concrete subtest, e.g.,

Specify tactile resolution on lower arm under varying conditions

Notice though what both have in common: specific & measurable

2/6/19 58

Source: “What makes a good research question?” http://twp.duke.edu/uploads/media_items/research-questions.original.pdf(Note: this is from humanities, where people are often less specific about the 2nd issue, i.e., “measureable”)

Examples for “specific”

1. Too broad and does not define the segments of the analysis:Why did the chicken cross the road?(The question does not address which chicken or which road.)

2. Too specific (could be answered with simple internet search):How many chickens crossed Broad Street in Durham, NC, on Feb 6, 2014?(Ostensibly, this question could be answered in one sentence and does not leave room for analysis. It could, however, become data for a larger argument.)

3. A more precise question might be the following:What are some of the environmental factors that occurred in Durham, NC between January and February 2014 that would cause chickens to cross Broad Street?(This question can lead to the author taking a stand on which factors are significant, and allows the writer to argue to what degree the results are beneficial or detrimental.)

http://twp.duke.edu/uploads/media_items/research-questions.original.pdf

Scientific method: Generally an iterative process ofa) observation & b) experimentation

Generally done by measuring data under varying conditions (with the goal to answer some research question, i.e., gain relevant scientific knowledge)

What are some of the environmental factors that occurred in Durham, NC between January and February 2014 that would cause chickens to cross Broad Street?

”Relevant knowledge”, i.e.:• Is the result of general value (bigger context & contribution)?• Is the result applicable to a general scenario (other conditions, etc.)?

Þ It is also important to be able to analyze the data with respect to the research question & goals


Measuring data under varying conditions (with the goal to answer some research question, i.e., gain relevant scientific knowledge)

What data to measure (& how)?

Quantitative, e.g., values that can be measured and expressed numerically• Different types exist (discrete/continuous, …)• Usually analyzed with statistical means

(averages, distributions, significance testing)

Qualitative, e.g., comments & observations• Can be subjective & biased• Not directly quantifiable, but means to measure exist, too,

e.g., categorize and encode for gaining general results

Note:• Separation not always strict (e.g., quantitative description of qualitative data

via questionnaires)• It can make sense to gather both (e.g., qualitative data to gather insight into

quantitative results, comparison of perceived versus actual performance, etc.)2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 60

How to gather qualitative data?

• Questionnaires• Formal or structured interviews• Cognitive walkthroughs (esp. in HCI)• Etc.

Example: user engagement

”User engagement is the emotional, cognitive, and behavioral connection that exits, at any point in time and possibly over time, between a user and a resource.”

From Attfield et al., 2011: Towards a science of user engagement

(position paper)

Research trend (esp. in HCI; in addition to usability)



How to measure?

Self-report:What: Happy, sad, enjoyment, …Means: Questionnaire, interview, think-aloud and think after

protocols, ...Attributes: Subjective, short- and long-term, lab & field, small scale

Physiology:What: Gaze, body heat, mouse movement, ...Means: EEG, SCL, fMRI, eye tracking, mouse-tracking, ...Attributes: Objective, short-term, lab & field, small & large scale

Analytics:What: Click, upload, read, comment, share, ...Means: Intra and inter-session metrics, data science, ...Attributes: Objective, short- and long-term, field, large scale



How to measure?

Self-report:What: Happy, sad, enjoyment, …Means: Questionnaire, interview, think-aloud and think after

protocols, ...Attributes: Subjective, short- and long-term, lab & field, small scale


Þ Qualitative measures

Established questionnaires often used in this context include:• Focused attention questionnaire [O’Brien & Toms, 2010]• PANAS questionnaire [Watson, Clark & Tellegen, 1988]


Focused attention questionnaire by O’Brien & Toms (2010)

1. I lost myself in this news tasks experience

2. I was so involved in my news tasks that I lost track of time

3. I blocked things out around me when I was completing the news tasks

4. When I was performing these news tasks, I lost track of the world around me

5. The time I spent performing these news tasks just slipped away

6. I was absorbed in my news tasks

7. During the news tasks experince I let myself go

Ratings on a 5-point Likert-scale from “strong disagree” to “strong agree”



PANAS questionnaire by Watson, Clark & Tellegen (1988)

“You feel this way right now, that is, at the present moment?”

Ratings on a 5-point Likert-scale with

1 = very slightly or not at all2 = a little3 = moderate4 = quite a bit5 = extremely

for the following 10 postive and 10 negative items (presented in randomized order):

• Distressed, upset, guilty, scared, hostile, irritable, ashamed, nervous, jittery, afraid

• Interested, excited, strong, enthusiastic, proud, alert, inspired, determined, attentive, active



PANAS questionnaire by Watson, Clark & Tellegen (1988)

Comments:

• Answers are used to calculate a mean positive affective score and a mean negative affective score (by summing up ratings for positive and negative items).

• The original paper verified the reliability and validity of this approach

• It provided a ground truth, i.e. for a “normal population”, the mean positive affective score should be 29.7 (SD = 7.9) and the negative affective score should be 14.8 (SD = 5.4)


General comments (not just for qualitative data):

• Be careful that you do not just end up gathering data

• Make sure you can gain some meaning and knowledge out of it!

For example:

• With no ground truth, what to compare against?

• If you have multiple conditions, are people able to distinguish them at the end of a long test session?

But if you ask them in between conditions, won’t that influence their later answers?


Measuring data under varying conditions (with the goal to answer some research question, i.e., gain relevant scientific knowledge)

What data to measure (& how)?

Quantitative, e.g., values that can be measured and expressed numerically• Different types exist (discrete/continuous, …)• Usually analyzed with statistical means

(averages, distributions, significance testing)

Qualitative, e.g., comments & observations• Can be subjective & biased• Not directly quantifiable, but means exist to,

e.g., categorize and encode for gaining general results

Note:• Separation not always strict (e.g., quantitative description of qualitative data

via questionnaires)• It can make sense to gather both (e.g., qualitative data to gather insight into

quantitative results, comparison of perceived versus actual performance, etc.)2/6/19 W. Hürst, http://people.cs.uu.nl/huerst 68

How to gather quantitative data?• Logging, recording, other measurements, …

The type of data usually suggets how, but be careful what you record, e.g., common mistakes:• Rounding errors, delays, ...• Do you have the right data?

(E.g., comparing averages requires comparable sample sizes)

What type you need to record depends on your research question, e.g.:• Independent variables (what you control)• Dependent variables (what you measure)Also: consider confounding factorsAnd: recording other data might help to gain further insight


Confounding factors

• Variables other than the independent variable that might affect the dependent variable

• Related to potential issues of your studies

(Note: they likely always exist, but try to minimize them and their potential influence)

Verify your experiment design with respect to:

• Reliability (the extend to which an experiment, test, or measure consistently yields the same result under similar conditions)

• Validity (the extend to which a concept, measurement, or conclusion is well founded and corresponds to real applications), in particular– Internal validity (is the relationship causal)– External validity (generalization of results)– Statistical validity (statistical relevance of conclusions)


A (bad) example

Assume a (real) museum with paintings & music playing (indep. variable).• Week 1: play classical music• Week 2: play rock musicMeasure average time spent in museum per visitor (dependent variable)

Result (e.g.): • In week 1, people spend on average 1.5 hours in the museum• In week 2, they spend on average 5 hours(Wrong) conclusion: play rock music because it attracts more people


How about:• Internal & external validity? E.g.: Comparable sample sizes?

Comparable subject characteristics (e.g., age)? Comparable conditions (e.g., tour busses with fixed schedule)?

• Confounding factors? E.g.: What if it rained in week 2?• Statistical validity? E.g.: Do the sample groups have the same size?

Are the results statistically significant?

Analyzing measured data: Statistical validity

What to report? E.g.:• Averages (also means, variance, …)• Distributions• Significance analysis resultsAgain, you are allowed to make compromises (if you can explain and justify them)

Also: what conclusions can you draw from the data?E.g.: Does spending more time in a museum with different music really suggest people spent time with the paintings (or did they just dance in the cafeteria?)

Note: correlation is not the same as causationAnd again: don’t just gather data, but meaningful data


Study design (with respect to the research question)

• Subjects; who, how many, …– How much do you need for statistical relevance?– How many samples per subject?– You are allowed to make compromises; but explain them

• Within/between subject design– Advantages between subject design:

less subjects, enables qualitative comparison– Advantages within subject design:

shorter test duration, no carry over effects

• Order & data/condition mapping, ...– Counterbalanced, Latin-square design, ...


Another example: Immersion in VR

Use multimodality to improve immersion in VR• Is it specific? (What is immersion anyway?)• Is it measurable? (and how?)

Hint: if you can’t measure it, maybe there is a correlation between a measurable unit? E.g., some research suggests that a higher perception of presence has a positive impact on enjoyment.

Example: Presence in VR

See survey paper by Schuemie et al. (2001), Research on Presence in Virtual Reality: A Survey. Cyber Psychology & Behavior, Vol. 4(2).


Schuemie et al. (2001), Research on Presence in Virtual Reality: A Survey. Cyber Psychology & Behavior, Vol. 4(2).

1. Introduction2. On the nature of presence3. Results of presence4. Measuring presence5. Causes of presence6. Discussion


2. On the nature of presence Various definitions exist, e.g.,• With respect to transportation, i.e., being “present” in the

virtual world• Presence (being in a computer generated world) versus

telepresence (being at a real remote location)• Personal (I am there) vs. social (other beings also exist there)

vs. environmental presence (environment reacts to me)• Subjective (likelihood of a person judging himself as being

physically there) vs. objective presence (likelihood of successfully completing a task)

Also, other distinctions, e.g.:• Immersion as an objective description of aspects of the

system (e.g., FOV) vs. presence as a subjective phenomenon (the sensation of being there); Slater & Wilbur (1997)


For your project: Specify concretely what you mean

3. Results of presenceTheories and empirical studies exist with respect to:• Subjective sensation of presence• Task performance• Responses and emotions• Simulator sicknessNot all of them are conclusive though.


For your project: Specify what concrete subdomain

you are addressing

4. Measuring presenceVarious approaches have been used in the literature:

• Subjective measures: questionnaires(note: various examples exist; see paper)

• Other subjective measures, e.g., continuous measure, presence counter, focus group exploration

• Objective measures: behavioral (e.g., reflex responses)• Objective measures: physiological (e.g., heart rate, skin

conductance, skin temperature)


For your project: Find suitable, reliable, feasible

means that fit your goal

5. Causes of presenceResearch devoted to finding factors that contribute to presence in VR.

Different categorizations exist, e.g., by Steuer (1992)

• Vividness (the ability of a technology to produce a sensorially rich mediated environment)

– Breath: number of sensory sensations

– Depth: resolution within each perceptual channel

• Interactivity (the degree to which the user of a medium can influence the form or content of the VR)

• User characteristics (individual differences in users)


2/6/19 80



HypothesisA prediction or best guess of the relation that exists

among the variables being investigated

Research questionIn quantitative research it is an interrogative sentence

that asks a question about the relation between two or more variables. In qualitative research, it is an

interrogative sentence that asks a question about some process, issue, or phenomenon to be explored.

Research topicThe broad subject matter area to be investigated Relate this to user engagement

• What is it (formal definition)?• What specifies it, or what

contributes to it?• How can we measure that?

For your project:• Be clear what you are

focusing on • Describe it formally• Pick the right sub-questions,

tasks, and appropriate means

That will make it easier:• To do it successful• To relate your (very specific

and focused) outcome to the “bigger picture” / major goal

For your project

2/6/19 81

What (general) problem are you approaching?What concrete sub-problem will you address?How are you going to verify your ideas/solutions?

Three slides (strict; plus one intro & one closing).

1. Problem (general goal and concrete sub problem)

2. Solution (your idea to address the sub problem and/or the research question you want to answer, related hypothesis, and objectives)

3. Methodology (your planned approach to verify your results or proof that your solution works)

Time limit is 5 minutes (& will be strictly enforced).

Recommendation for brainstorming: Try answering these questions

See also info on the website!

Project pitch (Feb 25):


“Soft” goals (for project):• Self-dependent work, e.g.

– Coming up with own ideas– Being able to specify them into research questions– Making realistic time planning and scheduling

• Teamwork

Learning goals / objectives of the course (project):

Projects: after successful completion of this part of the course ...• Students can apply basic techniques of multisensory interfaces to

create a multimodal interaction, e.g., for control of a game (demonstrated by the related project work).

• Students know the advantages and disadvantages of using different media types and modalities depending on different contexts, conditions, and applications (demonstrated by the related project work).

• Ideally/normally, you will get a team grade– Exceptions apply for particularly motivated or lazy students

– Keep a log book tracking your hours and activities!– You only have to report your total hours and major

contributions/activities in the end (see report template)– Log book is needed if above exceptions need to be applied

– Note: You won’t get punished for other people’s failures(but you might get punished for badly dealing with them)

– Better grades than the group grade are possible (e.g., for cum laude students), but YOU have to convince me why you deserve it

You find more info on the grading in the document about the project report (see website). Read it now!


Some comments on grading


Week 1Introduction

Papers 1, 5

Week 2 Lecture 1 (vision)Lecture 2 (sound, …) Deadline for forming groups





Deadline pitch slides: Feb 25, 9:00Deadline proj. plan: Feb 27, 23:59









Deadline proj. report: Apr 1, 23:59

Examweek

Exam

1. Read info on the website (all documents, including the slides that I skipped today; the ones under “HELPFUL LITERATURE” are optional, but might come in handy later)

2. Form groups (via spreadsheets); also specify there if you want to borrow our hardware

3. Brainstorm ideas while following the lectures & reading papers

4. Start preparing project pitches (for Feb 25)Note that they don’t have to be perfect and can contain open questions to ask the audience!

5. Start working on project plans (due Feb 27)You find a template and further info on the website.

And most importantly: have fun and enjoy the course!


Summary of next steps

If you have questions, email me ([email protected]).

2019 - infommmi - intro · week 8 analysis & report / paper writing week 9 project presentation...

Documents