tips for scale development: evaluating automatic personas
TRANSCRIPT
Automatic Persona Generation (APG)
News & Social Media Analytics Team
Social Computing GroupQatar Computing Research Institute
Hamad Bin Khalifa University
Information
architecture: Choosing
the correct information
elements and layout for
a given user or industry.
Comments: Finding
representative, relevant
and non-toxic comments
describing the persona.
Evaluation: Validating
accuracy, consistency,
and usefulness of
personas for individuals
and organizations.
Topics of interest:
Classifying topics of online
content and discovering
probable interests across
social media platforms.
APG: Platform for research
Description: Generating
fluent text descriptions of
the personas.
Discovering better ways to computationally process
and choose useful representations from vast
amounts of online data (”giving faces to data”).
Image: Using neural
networks to generate
persona profile pictures.
Story selection: predicting
and choosing content for
personas or content
creators.
Temporal analysis:
Observing change in
personas over time.
Information
architecture: Choosing
the correct information
elements and layout for
a given user or industry.
Comments: Finding
representative, relevant
and non-toxic comments
describing the persona.
Evaluation: Validating
accuracy, consistency,
and usefulness of
personas for individuals
and organizations.
Topics of interest:
Classifying topics of online
content and discovering
probable interests across
social media platforms.
APG: Platform for research
Description: Generating
fluent text descriptions of
the personas.
Discovering better ways to computationally process
and choose useful representations from vast
amounts of online data (”giving faces to data”).
Image: Using neural
networks to generate
persona profile pictures.
Story selection: predicting
and choosing content for
personas or content
creators.
Temporal analysis:
Observing change in
personas over time.
Validating personas
1. Personas as ”objective truth”
2. Personas as ”subjective truth”
For objective truth => accuracy metrics, predictive ability, stability analysis of personas
For subjective truth => case studies, ethnography, surveys
Validating personas
1. Personas as ”objective truth”
2. Personas as ”subjective truth”
For objective truth => accuracy metrics, predictive ability, stability analysis of personas
For subjective truth => case studies, ethnography, surveys
Persona Perception Scale:Toward systematic measurement of personas
Research Objective: Personas are notoriously difficult to validate and evaluate (Chapman & Milham, 2006). As fictive user representations based on real data, personas can be seen subject to interpretations from end-users of personas. These interpretations can be multi-dimensional and include elements such as empathy, perceived accuracy, and liking. We develop a persona perception scale and validate it among business professionals.
Chapman, C. N., & Milham, R. P. (2006). The Personas’ New Clothes: Methodological and Practical Arguments against a Popular Method. Proceedings of the Human Factors and
Ergonomics Society Annual Meeting, 50(5), 634–636.
Persona Perception Scale:Toward systematic measurement of personas
Plan: Develop the scale, validate with pilot study, and then conduct a large-scale survey for persona users. Once the scale has been published in journal, it can be referred to in other studies.
How does a system change affect persona perception?
Research process
1. Selecting dimensions and items => Literature review
2. Pruning them down => Expert feedback (contentvalidity)
3. Pruning them down => Pilot survey
4. Validating the scale => Large-scale survey (N > 300)
5. Publishing
6. Measuring impact of system changes to persona perception.
Research process
1. Selecting dimensions and items => Literature review
2. Pruning them down => Expert feedback (contentvalidity)
3. Pruning them down => Pilot survey
4. Validating the scale => Large-scale survey (N > 300)
5. Publishing
6. Measuring impact of system changes to persona perception.
Tips for scale development
Construct
Dimensions
Items
Remember: you’re trying use questions to reveal a hidden thing
you can’t directly observe
Remember: you’re trying use questions to reveal a hidden thing
you can’t directly observe
Persona Perception
Construct
Credibility
Consistency
Completeness
x1
x2
x3
x4
x5
x6
x7
Dimensions
Items
Tips for scale development
1. Start by collecting more items than you need in the end
2. Most sources seem to suggest a 7-point scale (but youcan use a 5-point scale as well)
3. Figuring out what you’re measuring is important (e.g., what is ’persona perception’?)
4. ”Would people answer these items in a similar way?” (simulate factor loadings in your head :)
5. Wording matters a lot…
1. Don’t mix positive and negative statements
For example,
”This persona seems useful”
”I don’t think this persona is very useful.”
Better is to have all the questions pointing at thesame ”direction”.
2. Avoid long sentences and complicated words
Even your mother should be able to answer with ease!
For example,
”I think I would have no trouble engaging in exhilaratingdiscussions with this particular persona.”
”I would enjoy talking to this persona.”
3. Avoid redundancy(=asking the same thing twice)
For example,
”The persona seems like a kind person.”
”I would see this persona as a kind individual.”
(However, the initial pool of items can have redundancy.)
4. Avoid combining two or more statements in one sentence
For example,
”This persona seems credible because it has the right picture and text content.”
”This persona has credible picture content.”
”This persona has credible text content.”
Checklist to consider for each construct/item
• Is it relevant? ✓
• Is it simple? ✓
• Is it clear? ✓
So that
1 = Not relevant
2 = Needs some revision
3 = Needs minor revision
4 = Relevant Can be used for calculating
content validity index (CVI)
Good constructs give you basis for hypothesis testing
E.g., structural equation modeling (SEM)
Persona Perception
Credibility
Consistency
Completeness
x1
x2
x3
x4
x5
x6
x7
Usefulness
Industry
Use case
Willingness to use
Too many tips to share…
A good reference book:
DeVellis, R. F. (2016). Scale Development: Theory and Applications (4 edition). Los Angeles: SAGE Publications, Inc.
Best is to try it out yourself!
And now… survey time!
Please go to: __________(also have print copies if someone doesn’t have a laptop.)
The beautiful team :)
Prof. Jim Jansen Dr. Jisun An Dr. Haewoon Kwak
Dr. Joni SalminenMSc. Soon-Gyo Jung