EVALUATION OF THEINTERNATIONAL WORKSHOP ON NEW COMPUTATIONALLY-ENABLED THEORETICAL MODELS TO SUPPORT HEALTH BEHAVIOR CHANGE AND MAINTENANCEOCTOBER 16-17, 2012, BRUSSELS, BELGIUM

Prepared by Donna Spruijt-Metz, Ilkka Korhonen, Stephen Intille, Wendy Nilsen, Niilo Saranummi and Heidi Lehtonen

Workshop background In October of 2012, 30 invited researchers from the United

States and Europe, 4 representatives from NSF, NIH and the European Commission, and 5 junior faculty/postdoctoral and pre-doctoral students met for 2 intensive days in Brussels.

Our task was to forge a path for major discoveries and innovations that would enable the use of existing and emerging technology to its full potential to understand human health behavior, change that behavior and maintain behavior change. The full mission statement is here:http://behaviorchange.be/index.php/mission/

The workshop was funded by NSF, NIH and the European Union, and led by Donna Spruijt-Metz, Stephen Intille, Niilo Saranummi and Ilkka Korhonen.

Evaulation This evaluation was conducted via

SurveyMonkey There were 11 questions total. 27 participants were kind enough to

complete the survey. The following slides summarize the

responses.

1. What is your main area of expertise/training?

(27 responses)

2. Prior to the workshop, what were your major goals or expectations for the workshop?

New, innovative solutions, strategies, ideas, future directions – thinking outside the box (8 people)

Multidisciplinary and multicultural views, meeting experts in different fields (8 people)

Learning more about state of the art, where we are now, key problems currently (7 people)

Meeting people, networking, new connections (7 people) Modeling behavior change; link between computer

models and behavior (6 people) Evaluation methods, new methods for testing theories

(2 people) Policy development; White paper; Intellectual stimulation,

challenge to existing ideas; Learn about technology; Learn about behavior change; Big data; Change the world (Other responses)

(27 responses)

3. Were your goals met?

The workshop exceeded my goals or expectations

The workshop met most of my goals or expectations

The workshop met some of my goals or expectations

The workshop did not meet my goals or expectations

44,4 %

40,7 %

14,8 %

0 %

Distilled comments: We spent much time on synchronizing our dictionaries and ways of thinkingIt takes time for the group to learn to communicate across disciplinesI don't think we made as much progress on thinking about how to evaluate such models and technologies based on themVery hands-on, very productive!We maybe need after this workshop a more pragmatic approach“the pace of the work was refreshingly aggressive” “amazing workshop, very well organized”“A great group of people!”

(27 responses)

Full comments Question 2: page 1of 2

Eagerly awaiting the white paper! I think that we spent most of our time on synchronizing our

dictionaries and ways of thinking, because it was a multi-disciplinary workshop. I think we need to obtain some sort of common language before we can actually reach the goal that I described above to satisfaction.

For the most part, the workshop more than met my expectations. By the end of the workshop, there seem to be a general consensus that we need better real-time individualized models of behavior change, and that control systems theory and agent-based modeling can help move us in that direction. I don't think we made as much progress on thinking about how to evaluate such models and technologies based on them, however.

It was just very hands-on, very productive! I think we had great discussions, but we need to continue to

have them. It takes time for the group to learn to communicate across disciplines. We made great progress.

Full comments Question 2: page 2 of 2

Overall, I was very satisfied with the workshop. I think it was excellent. It had been many years since I was in a

workshop so enriching The caliber of participant and the pace of the work were refreshingly

aggressive Great learning experience It was an amazing workshop, very well organized, that really exceeded

my expectations. The workshop was an excellent start We maybe need after this workshop a more pragmatic approach, trying

to provoke small changes, make them public (best practices, social networks,...) and after that, change the world :-)

I learned a lot and met some great people. I'm not sure I came out of it with a full understanding of what is the state of the art in behavior change, but I left the meeting with lots of next steps and reading to do.

A great group of people!

4. What were your most positive experiences (scientific, social, cultural or personal) at the workshop?

Multi/transdisciplinary experts, diversity of skills and expertise, many perspectives (13 people)

Meeting people, networking, community, new collaborations (7 people)

Arrangements (online pre-cooperation, provocations, 2nd day breakout groups, concrete goals, good sized group) (5 people)

Informal interaction, side conversations (3 people) New, provocative ideas (2 people) Firstbeat user testing ; Great primer on the state of

the science; That everyone was so responsive (others)

(23 responses)

5. On a scale from 1 (Poor) to 10 (Excellent), how would you rate each of the parts of the workshop?

1 2 3 4 5 6 7 8 9 100.0 % 0.0 % 3.8 % 0.0 %

11.5 %3.8 % 7.7 %

38.5 %

11.5 %23.1 %

The Idea Set and exchange of ideas beforehand

1 2 3 4 5 6 7 8 9 100.0 % 0.0 % 0.0 % 0.0 % 3.8 % 7.7 %

11.5 %

11.5 %

34.6 % 30.8

%Provocations

(27 responses)

5. On a scale from 1 (Poor) to 10 (Excellent), how would you rate each of the parts of the workshop?

1 2 3 4 5 6 7 8 9 100.0 % 0.0 % 0.0 %

3.7 % 7.4 % 7.4 %

22.2 %

22.2 % 14.8

%

22.2 %Breakout group 1 (Day 1)

1 2 3 4 5 6 7 8 9 100.0 % 0.0 % 0.0 % 0.0 % 3.7 % 3.7 %

11.1 %

25.9 %25.9 %29.6

%Breakout group 2 (Project development, Day 2 am)

(27 responses)

5. On a scale from 1 (Poor) to 10 (Excellent), how would you rate each of the parts of the workshop?

0.0 % 0.0 % 0.0 % 0.0 % 3.8 % 0.0 %

19.2 % 19.2 %

34.6 %

23.1 %

Breakout group 3 (White paper topic outline, Day 2 pm)

8 comments: A lot of time was used for developing a shared understanding of the terms used (2 people)“I missed the knowledge of professionals from the field of Communication, marketing and game industry””Groups seemed to run out of energy for final Breakout””We may have fallen short of concrete next steps””the breakout groups may have been too big to ensure everyone's voices were heard””especially pleased with the idea set responses and provocations”

(27 responses)

6. There were three main goals for the workshop. Do you think we achieved them?

(27 responses)

Goal 1. Identify strides that need to be made in innovation and scientific discovery related to understanding of human health behavior, behavior change, and behavior maintenance.

Missed Goal Achieved Goal Exceeded goal

3.7 %

92.6 %

3.7 %

6. There were three main goals for the workshop. Do you think we achieved them?

(27 responses)

Goal 2. Think of new ways to overcome barriers that may hinder this innovation that go beyond individual (siloed) research areas.

Missed Goal Achieved Goal Exceeded Goal

14.8 %

74.1 %

11.1 %

6. There were three main goals for the workshop. Do you think we achieved them?

(27 responses)

Goal 3. Identify major content areas that need to be addressed to truly move the science of human behavior change forward.

Missed Goal Achieved Goal Exceeded Goal

7.4 %

81.5 %

11.1 %

Comments from Question 6 “Useful but not always very clear, expect

a consolidated view in white paper” ”Not sure we had enough discussion on

ways to overcome barriers” ”We made strides but need more

communication to keep things going” ”Our work will depend upon generating

robust sets of co-occurrence data preferably with little effort and little cost.”

7. Was the appropriate expertise in the room to achieve our goals?

If you answered ‘no’ – what were we missing? More systems modelers (2

people) Communication, marketing More engineers/techies Folks from the smaller

companies and startups that are working in the mobile health/sensing space

More sensing-oriented folks, Data visualization and analytics, Architecture, urban planning, Neuroscience Behavioral geneticist

(26 responses)

Yes84,6 %

No15,4 %

8. Did your experience at the workshop impact your thinking or your (future) work?

Comments New partners and

collaborations (7 people) New areas to think about,

new perspectives, new ideas, new tools (6 people)

Experimentation and evaluation methods, study designs (4 people)

More understanding of other disciplines (4 people)

Reinforcement to current beliefs, validation of current work (2 people)

More focus on systems and modeling (2 people)

Yes

96,3 %

No3,7 %

(27 responses)

9. What would you like to see come out of this workshop? White paper, journal papers, report (16 people) Encouraging cross-disciplinary collaborations, understanding

of which players/disciplines needed (5 people) New Projects (4 people) A forum/community for connecting with others (3 people) New RFA (s) (3 people) Another conference/workshop (2 people) A working group/task force (2 people) Common data collection/repository (2 people) Metrics of success for academics (beyond

publications) ; Recommendations to EU-US policy makers; Cross-national comparisons; Educational incentives (fellowships, traineeships); Change in the way we all see and create systems for behavior change (others)

(26 responses)

10. Any ideas you would like to share for moving these ideas/this group forward in the future? We need a place where projects could be proposed,

commented on, refined; where people could work on projects together; a way to communicate (5 people)

Important to include the ”funding” side (policy-makers) (4 people)

Important to include the ”funding” side (policy-makers) (3 people)

A follow-up workshop, let’s meet again! (3 people) Set of specific projects, project ideas (3 people) How to bridge the theory, model and business,

Work on a common language across disciplines, write a paper together, a wish-list of resources and desired distal and proximal outcomes, What is being done by the organizers is very good (others)

(16 responses)

11. Is there anything else you would like to share about your experience at the workshop?

More time to get to know each other, to meet with people and interact (2 people)

What is the state-of-the art in computational modeling, more key note speeches would have guided the thinking

“Great people!” “Organizers did an amazing job!” “Organization of the workshop was perfect” “Such a wonderful experience” “Very good, very focused, very productive” “A wonderful and diverse group of individuals” “Networking sessions were amazing”

(15 responses)

Thank you all for sharing your thoughts!