Measuring User Satisfactionin Virtual Environment

Maciej A. OrzechowskiDesign System and Urban Planning Group

@ TU/e

Workshop Mass Customisation 26.06.2003

Plan

• Introduction• VR System (brief description)• Belief Networks (introduction)• Results of the experiment (Benchmark)• MuseV3 in action – Live Demo

The user is asked to modify that design according to his/her needs and desires.

General Idea ofMeasuring User’s Preferences

The Virtual Environment (VE) is used to present an architectural design to a user.

Behind that visual system there is a statistical model to estimate and predict respondent’s preferences based on applied modifications.

MuseV – VR System

MuseV3 – a virtual reality (VR) application with functionality of a simple CAD system for non-designers.

Two categories of modifications:• Structural modifications (change of layout)• Textural modifications (change of visual impression)

Structural Modifications

The most important from the point of view of estimation of user’s preferences.

Change of internal and external layout

Direct impact on overall costs

Expressed in simple and direct commands: create/resize/divide space; insert openings

Textural Modifications

Secondary modifications (visual impact), mainly used to check proportions, dimensions (inserting furniture) and to decorate (applying finishes).

Not included in the preference model

No influence on costs

MuseV3 in Desktop CAVE

Belief Network

Searching for new, flexible method to access user’s preferences.

Criteria:Criteria:• Interaction with the model during the time of preferences estimation

• Possibility to find weak points (where the knowledge about preferences is the worst)

• Improve data collection by direct feedback

• Incremental learning

Short explanation of BN

What it is?• Belief network (BN) also known as a Bayesian network or probabilistic causal network• BN captures believed relations (which may be uncertain, stochastic, or imprecise) between a set of variables which are relevant to some problem (e.g. coefficients and choices).

How does it work?After the belief network is constructed, it may be applied to a particular case. For each variable you know the value of, you enter that value into its node as a finding (also known as “evidence”). Then Netica does probabilistic inference to find beliefs for all the other variables.

Incremental learning.After the beliefs are found (post priori) MuseV updates the network, so they become a’ priori for the next respondent.

Step 0 Step 1 Step 5 Step 15 Step 64

BN - Model

In our proposal the network (model) is learning while a user is modifying a design!

To improve the quality of collected data and the knowledge about design attributes, the system, (based on beliefs), can post a question to user.

Experiment & Results

Experiment Types

There are in total four experiment types (FMVR, OEVR, MECA, VECA).

Two in each of two groups (VR and CA).

Each respondent had to complete two random tasks (one from each group), however each combination of tasks should be presented approximately equal number of times.

Experiment Types – cont.

VR Experiment CA Experiment

Type Free Modification Preset Options Multimedia Presentation

Verbal Description

Software(Mean of

presentation)

MuseV3 FM MuseV3 OE MuseV3 SC Web Pages

CollectionMethod

Interaction with 3D environment

Interaction with 3D environment

Questionnaire Questionnaire

Task Modification of architectural

design

Respond to pre designed options

Choice from between three

design alternatives

Choice from between three

design alternatives

Interactivity with 3D model

Restrained to design constrains

Finishes and furniture

Walk Through N/a

Feedback from the system

yes yes none none

Estimation method

Belief Network Belief Network MNL Model MNL Model

Results

RespondentsThe truth about the respondents:

We sent 1,600 letters in total !!!!

The preparations to send those letters took 2,5 daysfor two people (Vincent and Maciek)

Within 2 weeks we received 96 positive conformations.

At the end of the experiment we end up with solid number of 64 respondents that have completed the both appointed to them tasks!

5 of the 64 respondents would not buy the house that they have designed.

4 respondents did not completed second task(as the design was not relevant to them)

2 respondents did not started the experiment for the same reason!

The most preferred system

The difficultness and pleasure of the tasks.

External validity Real Life Data – Overall (CA, BN)

G-O-F of REAL LIFE (RL) PREDICTION (Rho2 calculation based on log likelihood)Calculations based on BETAS

GOF (CA) = -0.0384GOF (BN) = 0.1128

External validity Real Life Data – BN (FMVR, OEVR)

G-O-F of REAL LIFE (RL) PREDICTION (Rho2 calculation based on log likelihood)Calculations based on BETAS

GOF (FMVR) = 0.13096GOF (OEVR) = 0.01979

External validity Real Data – BNbased on probability distribution of each option

Option Type

Lounge Ext.

First Floor Ext.

Garage Ext.

Extra Kitchen

Bedrooms Dormer Window

Real life situation 0.867 0.867 0.667 0.067 0.067 0.133

Belief Network includes both subtypes (all respondents)

0.81 0.35 0.71 0.32 0.208 0.519

Subtype: Free Modification 0.83 0.334 0.781 0.491 0.168 0.391

Subtype: Preset Options 0.81 0.394 0.633 0.156 0.244 0.666

The table illustrates ratio (percentage) of choosing certain design option.In case of real life - based on numbers of subjects buying certain option.Ri = Ni / N, where Ri – ratio for option i, Ni – number of subjects choosing option i, N – all subjects

In case of BN based on beliefs read from the network.

Summary• The majority of the respondents prefer the VR environment to the traditional.

• Respondents highly valued the freedom in modifying the architectural design.

• Due to learning and understanding the software - VR is slightly difficult.

• The traditional method was find as the most difficult (due to problems related with imagining the description of the house)

Summary• Direct observation of respondent's engagement (created designs and the time spent on the process) into the VR - indicates that people prefer to work with 3D models rather then with textual description.

• The possibility of experiencing with the not existing house reinsure users' decision, raise questions and provokes discussions.

• The numerical analyses showed that working with virtual reality helps respondents to understand the design and improve their decision consistency.

MuseV3 in Action !DEMO