report on 2008 debating science program survey science program/imx... · 2018-01-05 · debating...

Debating Science 2008 Survey Report, 8/17/09

1

Report on 2008 Debating Science Program Survey

August 17, 2009

Compiled and summarized by:

Renee Madathil, B.A.1

Blake Francis, B.A.2

David Schuldberg, Ph.D.3

Contact Information:

Department of Psychology

The University of Montana

Missoula, MT. 59812-1584

406-243-4521

[email protected]

[email protected]

[email protected]

1 Department of Psychology, The University of Montana

2 Department of Philosophy and The Center for Ethics, The University of Montana

3 Department of Psychology, The University of Montana


2

Executive Summary

This survey report comprises a summary of responses from the summer 2008 Debating

Science Program, A New Model for Ethics Education. The New Model for Ethics Education

involves an approach which guides deliberation through a logical structure designed to foster

informed and thoughtful discussion. The Debating Science Program has two connected goals: to

educate students about the scientific, ethical, and social issues involved in science debates and to

teach students to develop habits of ethical deliberation. The weeklong workshop in Montana

provided the students with the opportunity to think about and discuss scientific, ethical and social

issues through attending expert lectures on numerous topics and small group sessions based on

interest. During the semester long online course students participated in deliberation on the

“Online Deliberation Center.” The ODC guides deliberation using a logical structure by asking

students to consider the social goals, alternatives, obstacles, and side effects. This survey is

intended to evaluate the extent to which the Debating Science program has met these goals.

Prior to the summer workshop, participants expressed general agreement about

deliberation and science policy as well as the quality of public science debates. Several

respondents rated the quality of current public debates over science policy as „bad.” In

elaborating this response participants discussed the public‟s low level of science education and

the poor quality of the scientific information available to the public. Several others mentioned

the nature of science, and expressed concern over the capacity of scientists to engage the public.

Overall, the majority of respondents reported that scientists should play a role in science debates

and the vast majority envisioned being involved in public policy deliberation during their

careers. Most participants rated their preparedness to enter into public debates about science


3

policy with multiple stakeholders as neutral (44%) or good (35%). The majority of respondents

indicated that they were not involved at all as an active participant in public debates, while

several (42%) indicated they were somewhat involved.

After the workshop, students reported having a positive experience. Participants

mentioned the lectures, the quality of the speakers, the range of lecture topics, and the

effectiveness of group work. With respect to group work, specifically, several respondents

mentioned the benefits of working with a diverse group of students. Suggested improvements

included more time for discussion, the need for introductory science lectures, more in depth

science lectures, and better defined goals for group sessions. The workshop had a positive effect

in raising student awareness of the social and ethical complexities associated with emerging

technologies, on student confidence in participating in interdisciplinary ethical dialogue, and the

likelihood of participants getting involved in public policy deliberations in their career.

Participants rated the workshop overall as excellent or good.

Reflecting on the entire program, participants responded that online discussion and chats,

collaborative experiences, the summer workshop, instructor interaction, and deliberative

opportunities were among the strengths of the course as a whole. Descriptions of the course‟s

weaknesses were similarly varied, including content, format, goals, and participation. Some

participants reported having difficulty with the wiki, mostly having to do with format and other

barriers to participation. Overall, respondents found the online course to be a positive experience

and agreed that the course allowed them to place the scientific research discussed into the

broader social and ethical contexts. Participants responded favorably to interdisciplinary

deliberations. The majority of respondents felt comfortable exchanging ideas and opinions on the

ODC and felt it had potential as a tool for discussion for experts and graduate students, although


4

this measure seemed to vary depending on the course. Participants in the climate change and

nanotechnology groups provided mixed responses regarding their experience deliberating on the

ODC. Several respondents from these groups cited discomfort in online participation and lack of

deliberative discussions while online. The biotechnology group reported an overall positive

experience. Respondents from this group reported feeling comfortable participating and provided

detailed descriptions of online deliberation.

Introduction

The Debating Science program, a New Model for Ethics Education, is intended to teach

prospective scientists how to become more effective participants in public science debates.

Debating Science, conceptualized as a pilot project, completed its second year with the 2008

program. The 2008 Debating Science program consisted of a five day face-to-face workshop at

the University of Montana, Missoula during the first week of August 2008 followed by a

semester long online course during fall 2008. The face-face-workshop consisted of lectures,

group sessions, training sessions on the Online Deliberation Center, and field trips. Twenty-eight

students participated. The students were broken into three groups by interest: Climate Change,

Nanotechnology, and Biotechnology. Students were selected to participate in the groups through

a competitive application process, and were placed into groups such that each group represented

a diversity of opinions and academic interests. There were 10 students in the Biotechnology

groups, 9 in Climate Change, and 9 in Nanotechnology. A separate instructor led each group

session. All twenty-eight students attended lectures, which covered various scientific and

technical components as well as social issues of each topic. Several experts from each of the

three fields were invited to give lectures and to participate in relevant group sessions.


5

The online portion of the program consisted of student participation in the Online

Deliberation Center (ODC). The ODC is intended to facilitate deliberation online by allowing

students to collaboratively construct an online document and discuss their work online. During

the semester course, the students engaged with their respective group and instructor.

Communication was facilitated by email, the ODC, and live chat sessions.

Survey Methods

Internet-based surveys of the participants were conducted prior to the workshop,

immediately after it, and at the end of the following semester. For the purposes of this report,

surveys were divided by dates of administration, and information is reported separately for each.

Qualitative data were analyzed and summarized, and this information is presented first for each

survey.

Participants

Demographic information on participants was gathered at the first (pre-workshop) survey

administration. Of the 25 participants who responded to Item 27, 84% identified themselves as

“White.” Of the 25 respondents to Item 26, 56% were female, and 44% were male. Most

participants had taken previous ethics courses at varying degrees and requirement levels. The

most commonly reported career choice of the participants was academia. Tables summarizing

demographic information follow:


6

Table 1

27. Please indicate the racial and/or ethnic category with which you identify. Enter as many

categories as you wish.

Response Total Response %

African American or Black 0 0%

American Indian or Alaskan Native 0 0%

Asian 3 12%

Native Hawaiian or Other Pacific Islander 0 0%

Hispanic 2 8%

White 21 84%

Other, please specify 0 0%

Total Respondents 25

(skipped this question) 2

Results

Pre-Workshop Survey (Qualitative)

Participants in this survey reported several qualities essential to deliberation about

science policy. The most frequent responses were: critical thinking, open mindedness, and

preexisting scientific knowledge. One respondent made the following remark: “To engage in

deliberations about science policy a person needs some understanding of the science involved.

They should be curious, honest, articulate, critical thinkers.” One participant noted that although

open mindedness was important, one should be firm in their stance, stating that he or she should

not be “easy to influence.”

Knowledge of scientific issues among the public was seen as poor. Several respondents

maintained that Americans were poorly educated in matters of science policies. One respondent

made the following remark: “…most of the public does not understand the language, culture, or

conventions of science.” This respondent went on to describe this in terms of the media,


7

claiming that they could “drive the debate by reporting on issues in ways that seem unbiased but

in fact are not.” Another respondent suggested that misunderstanding exists in both among the

public and among scientists, stating that there is a “lack of understanding regarding the nature of

science by the general public [and a] lack of understanding of the general public by scientists.”

Several individuals responded that debates about science policies are not true debates;

rather, they are an expression of opinions. One participant stated, “Science policy debates are

approached as political debates, and hence as a matter of ideology or mere opinion.” Others

talked about potential political motivations behind scientific debates. For example, one

individual stated, “People seem obligated to identify with causes based on their political,

religious, or social ideology—not on actual scientific information.” Further, other respondents

held that there is a sense of rigidity in current ideologies due to political or religious affiliations.

Most individuals listed climate change first in response to the prompt, “List two of these

social and environmental challenges, if any, in the order of their importance” (Table, 10, Item

13). There were a variety of responses for the second challenges, including energy management,

lack of access to basic resources such as water, and limited resources.

Several individuals indicated that the role of scientists in addressing these challenges is

the education of public and policy makers. For example, one individual wrote, “Education is the

key to conveying that there are issues. Scientists must foster education of the issues as well as

general education so that society may better deal with the challenges.” Another individual also

expressed a similar belief, “…There seems to be a gap between the scientific knowledge and the

beliefs in the „real world.‟ We need to figure this out and find a way to convey the necessary

information in a credible way.” Other respondents observed the “ineffective” nature of means of

communication between scientists and the general public. One respondent noted, “The way that


8

scientists are taught to communicate is ineffective. Even for applied science, the general scheme

is to perform experiment[s], [and then] publish technical paper[s] in a technical journal.” Other

participants held that scientists should adapt their language in order to be more “meaningful” to

the public.

In regard to career plans, most respondents indicated that they were interested in

academia. Several also stated that they would like to be involved in research, either at a

university, or at a research institution. Finally, some respondents expressed a desire to complete

a post-doc upon graduation.

Most respondents indicated that learning about proper language and terminology would

be most helpful to better prepare to participate in policy issues (Item 24). For example, one

respondent stated, “It would be especially helpful to learn about the methods, terminology, etc.

of science policy professionals.” Other respondents expressed a desire for practice, stating,

“Broader foundation in philosophical considerations, grounded in case studies and hypothetical

examples” and “Real world experience!” As discussed in earlier responses, some participants

reported a need for a more “accessible” communication style when interacting with the public.

One respondent stated, “I would like to have a better understanding of how to communicate my

own and others‟ scientific research in a way that is accessible and engaging. I am especially

concerned with how to talk about complexity and uncertainty in a way that does not detract from

the power of what we know right now.” Other respondents commented on a desire to have a

better understanding of issues in science policy. For example, one respondent stated, “…I feel

that a strong introduction to the policy issues behind scientific research requires a detailed

examination of the requisite background of the relevant science in question…I would then like to

be introduced to the specific character of the policy issues at hand including an explanation of


9

the political framework through which the debate has taken place (specifics on legislation

governing the research, funding, etc.).” Similarly, another respondent also expressed a desire for

“a general understanding of how it all works.”

Pre-Workshop Survey (Quantitative)

The following tables summarize the quantitative data gathered during the Pre-Workshop Survey.

This survey was administered prior to participants‟ arrival in Montana for the face-to-face

workshop. The survey inquired into both the participants‟ familiarity with the subject matter of

the workshop, but also their understanding the scientists‟ role in ethics and public deliberation

and interdisciplinary ethical dialogue.

Table 2

Table 3

4. How confident are you about participating in interdisicplinary ethical dialogue at

your university?

Extremely

confident

Somewhat

confident

Neither

confident

nor unsure

Somewhat

unsure

Absolutely

terrified

Interdisciplinary 17% (4) 46% (11) 33% (8) 4% (1) 0% (0)


(Skipped this

question) 3

3. How satisfied are you with your training in graduate school to date in the

intellectual qualities you mention above?

Very

Satisfied Satisfied Neutral Dissatisfied

Very

Dissatisfied

Satisfaction Level 12.5%(3) 37.5%(9) 45.83%(11) 4.17% (1) 0% (0)


(Skipped this

question) 3


10

Table 4

5. How would you assess your current level of preparation to enter into public debates

about science policy with multiple stakeholders?

Very

Good Good Neutral Bad Very Bad

Preparation 4.35%(1) 34.78%(8) 43.48%(10) 17.39%(4) 0% (0)


(skipped this

question) 4

Table 5

6. Scientists should be trained in research ethics (e.g. how to present data, how not to

plagiarize).

Strongly

Agree Agree Neutral Disagree

Strongly

Disagree

Research ethics 84.62%(22) 15.38% (4) 0% (0) 0% (0) 0% (0)


(skipped this

question) 1

Table 6

7. Scientists should additionally be trained in social and environmental ethics (e.g.

ethical obligations to people and to nature).

Strongly


Strongly

Disagree

Social and

environmental ethics 61.54% (16) 34.61% (9) 3.85% (1) 0% (0) 0% (0)


(skipped this

question) 1


11

Table 7

8. How would you rate the quality of current public debates over science policy?

Very Good Good Neutral Bad Very Bad

Quality 0% (0) 11.54%(3) 30.77% (8) 50% (13) 7.69% (2)


(skipped this

question) 1

Table 8

10. To what extent do you believe that it is good for scientists to participate in public

deliberations about directions for science and technology research?


Participation 57.69%(15) 42.31%(11) 0% (0) 0% (0) 0% (0)


(skipped this

question) 1

Table 9

11. How much influence should non-specialist citizens have over the ultimate direction of

scientific research?

Very Small Small Neutral Large Very Large

Direction 0% (0) 19.23%(5) 61.54%(16) 19.23%(5) 0% (0)


(skipped this

question) 1


12

Table 10

12. Do you agree that global society faces serious social and environmental challenges?

Strongly


Strongly

Disagree

Serious challenges 88% (22) 12% (3) 0% (0) 0% (0) 0% (0)


(skipped this

question) 2

Table 11

14. Solving such global challenges requires solutions that are:

Primarily

technological

Mostly

technological

but also political

Mostly political

but also

technological

Primarily

political

Solutions 8% (2) 36% (9) 68% (17) 4% (1)



Table 12

16. What is your level of understanding of the term "food security"?


Food Security 10.52% (2) 47.36%(9) 31.58% (6) 5.26% (1) 5.26% (1)


(skipped this

question) 8

Table 13

17. What is your level of understanding of the problem of creating a "nano-divide"?


Nano-divide 0% (0) 17.65%(3) 23.53% (4) 29.41%(5) 35.29%(6)



13

(skipped this

question) 10

Table 14

18. What is your level of understanding of the phrase a "fair and effective response" to

climate change?


Climate response 11.11% (2) 44.44%(8) 33.33% (6) 0% (0) 11.11%(2)


(skipped this

question) 9

Table 15

19. What is your level of understanding of the issues raised by the "precautionary

principle"?


Precautionary

Principle 23.08% (6)

30.77%

(8) 23.08% (6)

11.54%

(3)

11.54%

(3)


(skipped this

question) 1

Table 16

20. What is your level of understanding of the term "distributive justice"?


Justice 15.38% (4) 26.92% (7) 26.92% (7) 15.38% (4) 15.38% (4)


(skipped this

question) 1


14

Table 17

22. Are you currently an active participant in public debates over science and technology?

Very involved Somewhat involved Not involved at all

Current involvement 7.69% (2) 42.31% (11) 50% (13)


(skipped this

question) 1

Table 18

23. Do you anticipate being involved in any public policy deliberations in your chosen

career?

Very Likely Likely Neutral Unlikely

Very

Unlikely

Expected involvement 42.31% (11) 42.31% (11) 15.38% (4) 0% (0) 0% (0)


(skipped this

question) 1

Post-Workshop Survey (Qualitative)

When asked to comment on any aspect of the workshop that was particularly strong, most

of the 22 respondents mentioned the quality of the lectures. Several respondents found the

lectures to be strong in general. One participant remarked, “The whole workshop was great. I

think the lectures were very interesting and well balanced among the three areas

(nanotechnology; climate change, and biotech). The organizers did a great job finding top of the

shelf speakers who addressed the different aspects of these areas.” Another participant remarked

on the “caliber” of the speakers and added, “The addition of journalists to the mix was great

too.” Several lecturers were brought up favorably and mentioned by name; these included: Dan

Fagre, Juliet Elperin, Don Brown, Christopher Preston, Dane Scott and Andrew Light. A

participant said of Andrew Light that he “gave what I found to be the most compelling argument


15

for scientists to be involved in policy debates around controversial issues related to science.” A

few respondents qualified their comments regarding the lectures. One mentioned, “It would have

been nice to have a more basic intro to the science for some of the aspects.” And another noted

that although “Most of the speakers where pretty good,” there were “a few misses in my

opinion.”

Several participants made positive remarks about seminars and group work. Several

participants enjoyed the “time to interact with…peers.” For instance, one remarked, “Even

within each group there was a range of backgrounds which opened my views and thinking

beyond my research work and my concerns.” Similarly, another respondent stated, “Seminars

helped in learning deliberation.” One respondent noted the mixed views within the seminars:

“[In] the individual seminars…one got an overall view of the problems and also heard both the

scientists‟ as well as the philosopher‟s view of the issue.” Other participants indicated that peer

interaction in general, not necessarily in organized aspects of the course, was beneficial to them.

One remarked, “The team building was excellent in the biotech group.” Another participant

stated, “The best part was just meeting people who are interested in how science relates to

society and vice versa. I loved being able to just use each other as sounding boards; learning new

ideas from each other; learning how to better communicate our ideas on different issues.”

Most participants also indicated that time management was particularly strong. “Time

management was good. There was never a moment to feel bored or disengaged or tired. I

enjoyed the fact that we did not have to waste time looking for food during lunch time.” Many

respondents remarked that the days were “busy.” Several responses also stated the strength of the

extra-curricular activities, explicitly mentioning, the hike, and the PEAS farm.


16

When asked to comment on any aspect of the workshop that needed improvement,

several participants felt that the lectures could have been improved. Many participants mention

that lectures did not provide enough time for discussion. For example, “I…felt that there were

several lectures that would have been better if they had less time...This would have left more

time for questions; which was lacking during some presentations.” More specifically, one

participant noted, “I would have liked more time at the end of lectures for discussion. A number

of times it felt like the discussion was cut off just as it started going. I appreciated the willingness

of speakers to stay after the talk and speak informally with folks; but often it felt like that was

were the really meaty discussions happened; and it would have been nice to have more people at

least witness those conversations.”

Other respondents found the lectures less rewarding. Some felt a lack of interest in

lectures that were “outside my home group.” And some felt that the lectures were lacking in

content, for example, “I think we heard four different introductions to nanotechnology; but very

little substance or depth.” This participant also added, “It would have been more productive to

have more group meeting time and less attending of lectures outside of one‟s interest area.”

When it came to group meetings, a few participants saw room for improvement. One participant

felt that “better definition of goals and intent up front” was needed. A different respondent felt

that “some objective [goals] should be set up on [the] first day.” And another participant

suggested along similar lines, “Perhaps it would have been more productive to have more time

within the individual sections to really delve into issues.” Overall, many respondents felt that

more time was needed for discussions in both the lectures and the seminars.

When it came to the logistics of the trip, two participants mentioned that pre-arrival

planning was wanting. Several participants mentioned the living situation, specifically lack of


17

wireless connection in the dorm room and hard mattresses. Four participants commented on the

quality of the lunches, asking for more variety and more vegetarian options.

Descriptions of types of training would be most helpful to prepare to participate in policy

issues varied. First, many descriptions included discussion of policy, media, and

communications. Several participants discussed the need for training in matters of policy. A

participant stated, “having an understanding of factors involved in policy issues; a greater

awareness; knowing the lingo; and actual practice in deliberation helps greatly.” One respondent

thought, “More interaction with actual policy makers would be beneficial.” This particular

respondent added the importance of examining the “current political climate” in order to “couch

ethical arguments within a greater socio-political context.” Another respondent was interested in

understanding “how scientists‟ information is actually taken and used in policy.” Similarly,

respondents were interested in training with respect to “dealing with the press.” Two respondents

cited Juliet Elperin‟s mock interview exercise, “as useful in terms of dealing with the media.”

Both respondents were interested in more “hands on training.” Finally, a few respondents

mentioned training in rhetoric, or “how to express your opinions.” One sated that “Learning how

to actually communicate issues from a structured framework would be great.” Respondents also

focused on training in “writing in a more appropriate manner.”

Second, several respondents made more specific comments concerning their need for

training in terms of the workshop itself. One respondent commented on the “wide berth” of

investigation in the workshop and stated that s/he would have preferred “specific case studies

with clear outcomes.” S/he continued, “The activities planned during the week could have been

more constructive rather than revolving mostly around instruction.” Another respondent wanted

“More information on how to get involved and/or speak about issues at a local level.” And


18

another respondent wished “we would have had more designated time at the end of every topic

day to discuss issues with all participants—like a round table discussion or debriefing.”

Finally, two participants stated that they need training in deliberation. One stated, “I do

not feel fully prepared to debate the ethical issues related to my work. I would like to see more of

the fundamental skill sets necessary to deliberate meaningfully.” And another listed, “Learn how

to deliberate; [that] is bring to the table the very best of every participant in the deliberation.”

Post-Workshop Survey (Quantitative)

The following tables summarize the survey responses for the survey administered after

the workshop. The information presented represents both the participant‟s evaluation of the

workshop itself as well as the participant‟s confidence in participating in interdisciplinary

dialogue.

Table 19

1. What is your overall evaluation of the workshop?

Excellent Good Neutral Fair Poor

Overall 59.1% (13) 40.9% (9) 0% (0) 0% (0) 0% (0)


Table 20

4. The workshop raised my awareness of the social and ethical complexities associated with

emerging technologies.

Strongly


Strongly

Disagree

Complexities 50% (11) 45.45% (10) 4.55% (1) 0% (0) 0% (0)


Table 21

5. The workshop increased my confidence about participating in interdisicplinary ethical

dialogue.

Strongly


Strongly

Disagree

Confidence 31.82% (7) 54.55% (12) 13.64% (3) 0% (0) 0% (0)


19


Table 22

7. As a result of this workshop, do you think it is more or less likely that you will get involved

in public policy deliberations in your chosen career?

More Likely About the same Less Likely

Expected Involvement 77.27% (17) 22.73% (5) 0% (0)


End-of-Semester Survey (Qualitative)

The End-of-Semester survey covered the semester long online course as well as the

Debating Science program as a whole, including the face-to-face workshop in Missoula during

the summer. During the semester course the students were broken into groups according to

subject matter (biotechnology, nanotechnology, or climate change) and assigned to an individual

instructor. For this reason, qualitative questions that specifically deal with the online deliberation

component of the program will be analyzed separately according to subject matter

(biotechnology, nanotechnology or climate change). The general questions will be treated first.

Analysis of All Groups

Respondent‟s descriptions of what part of the course worked best for them were varied,

but several respondents mentioned the online discussions and chats. For example, “The group

chats were …good for touching base and discussing the issues we were working on in the ODC.”

Another participant remarked, “The online chats we had definitely made me more productive on

the ODC the weeks we had them.” Several respondents mentioned the ODC positively.

However, some found the ODC difficult, “I found the asynchronous communication of the wiki

not very user friendly.”


20

Similarly, a few participants mentioned that collaboration online was the best aspect of

the course. One respondent remarked, “The collaboration on the writing of the final pages was a

wonderful experience. It was a great learning experience to collaborate with researchers from

different disciplines.” Another respondent noted, “It was good to have collaborative teams.

Accountability seems to have been our biggest barrier; and this helped to alleviate that.” Several

participants described the face-to-face workshop in Missoula as well. “The workshop in

Missoula helped tremendously to form personal bonds; which led to a, in my eyes, very

productive deliberation. The various backgrounds of the students with regard to the topic led to a

good exchange of opinions and growth of personal views.” And another remarked, “Meeting

fellow participants in Missoula was by far the most fulfilling aspect of the course.”

Some respondents described course content and instructor interaction as working the best

in the course. One participant commented, “I…thought that the selected readings were well

chosen and that the assignments derived from the readings were directly relevant to course goals.

When it came to faculty interaction one respondent mentioned an instructor by name, “Regular

contact with Prof. Scott; his evaluations and comments was …[an] important aspect of this

course.”

When participants described the parts of the course that worked least, comments ranged

widely. However, most comments concerned the format, content, goals, and participation rather

than the technological components of the course. Several participants commented that the course

“lacked focus” or that they were unsure about the content of the course. One respondent noted,

“The lack of directed focus stunted the growth of our project and because of a lack in strong

central focus; our group‟s goals kept changing as the weeks went by without much progress.”

Another participant felt that the course “branched out too far.” One participant was concerned


21

with the direction of the project generally, stating, “I felt like we were mostly handed an

ambiguous challenge and told to figure it out. Without addressing some of the underlying drivers

behind why we are working on these issues; we are doomed for failure and an utterly incomplete

understanding.” This person would have preferred “a more fundamental approach” and also

commented that “It was very difficult to determine the level of detail required.” Along the same

lines, some participants felt that the content of the assigned readings detracted from the course.

One respondent remarked, “…the focus on describing the climate change mitigation wedges

often felt like busy work. It didn‟t help me learn anything about the ethical dimensions of the

climate change debate; and instead I felt like I spent long hours researching and writing about

topics that have been extensively described elsewhere.” Another respondent added, “The short

discussion question about the readings often felt like busy work and often distracted from the

course‟s goal of working on the main ODC pages.”

Several participants brought up participation as an issue. “The course took a long time to

gain momentum; as many participants did not respond to readings and assignments early on.”

Also, many respondents noted that the course lacked “equal participation.” Similarly,

participants commented on the interactions with other participants. Likewise, another participant

struggled with “Coming to a conclusion…unanimously for such a vast topic with so many people

putting their heads together.”

Some participants noted they had difficulty participating on the wiki. For example, “The

editing process for the actual ODC pages was kind of tricky. Even though we were told to edit

freely, it‟s easier said than done. It helped that we all knew each other and got along well; but it‟s

still difficult to go through someone else‟s work and make changes.” Other participants had more

difficulty with the ODC. For example, a participant noted that “Getting people to work together


22

in this format” was the part of the course that worked least. One commented, “It was easy to get

behind in work and when I would log on to see the changes it was difficult to compare. The more

sections that were written; the more difficult it was to make changes because you wouldn‟t want

to delete another person‟s work without good reason and modifying it became tricky.” Another

person had more technical difficulties, “…the site lacked many things. I never got emails when

people were active; so the only way to engage was by going to the site. You couldn‟t cut and past

from the wiki to word—you could only cut and paste from syntax frame. This made the site

much less user friendly.” Lastly, one participant felt “daunted” when faced with contributing, “it

was somewhat daunting to read through everyone‟s response before composing my own

response feeling like what I was going to write was lacking in sophistication or depth.”

When asked what types of training would be most helpful to feel better prepared to

participate in policy issues the majority of respondents discussed the need for ethics and policy

training, with some more focused on training in media and in science. One respondent said

“Ethics would be very important because I don‟t really think one should be out in the public

realm engaging in these issues without having developed a well thought out position on the

issue…I would hope that people‟s positions are the result of sound reasoning within their chosen

ethical framework.” This respondent went on to discuss training in policy and political theory.

Another participant suggested, “Ethics taught in graduate school for science majors would be

helpful; and not research conduct ethics as it‟s taught now.” Several other respondents pointed to

needed training in policy, political theory, science, and economics.

Respondents also noted that more training is needed in communication with media. One

noted, “I think I have a lot of room for improvement in how I interact with media and the press.”

Another participant noted the need for “practice and training to feel more confident and


23

comfortable participating in panel discussions [and] being interviewed.” Other respondents

discussed leading or facilitating public deliberation sessions. One participant noted, “The public

is rather…polar when it comes to discuss policies in our subject matters…The public needs to be

better prepared for deliberations and discussions.”

Several participants responded to the question pertaining to whether there is anything

helpful they would like to share. A few participants said that they had a positive experience. One

respondent stated, “I though this was a great course. The workshop was absolutely indispensable;

I think it would be very difficult and much less pleasant to try and carry out the same project

without at least some amount of face-time.” Another participant suggested, “It would have been

helpful to have some online chats with experts during the course.” At the same time a few

participants expressed concerns about the goals of the course. One stated, “…I think you need to

rethink your structural approach. It‟s a challenge to teach a generation how to think about these

problems; and I think you have failed at this. You gave us a forum to collaborate on research; but

you did not teach us anything about how or why to think about the implications.” Similarly,

another respondent remarked more specifically, “A more structured system of goals and

deadlines would have been helpful.” A couple of respondents commented that the program

needed to work on “group dynamics.” For example, “I became very disconnected from the group

after about four weeks of crawling progress due to incessant bickering about the particulars of

the project which I felt mainly came from some who kept pursuing an agenda that was not shared

by the group.” This participant went on to say, “There just came a point where I felt totally

unnecessary and only had minimal responsibilities; which lead me to become somewhat

disenchanted with the group and; by proxy; with the project.”

Analysis by Section


24

Six questions are analyzed according to section. Each of these questions relates to the

online class deliberations in which students were placed into sections according to their field of

interest. It is important to note there were an uneven number of respondents to each of the

questions between the groups.

Climate Change

Five participants from the Climate Change group responded to the question concerning

the effectiveness of the instructor, a follow up to Item 6 (Table 26). Two respondents answered

positively overall, stating “Becks did a great job of keeping us focused and on track” and “Becks

did a good job of keeping the project on task by checking up weekly, offering suggestions, and

focusing us when the work became inefficient or distracting.” One respondent expressed

frustration concerning the website. Another respondent said, “I just don‟t care for chatting;

especially when we are not speaking the same language.”

When asked to discuss whether their positions changed through deliberations with other

students, four of the respondents from the climate change group varied. One participant

answered overwhelmingly positively, stating, “Yes, I learned a lot from the perspectives shared

with me by other students…Whereas my initial perspective was to deprecate wedges, discussion

with fellow students convince me of their efficacy.” Another participant noted that “some of my

positions changed….I was surprised at some of the results we came up with.” And two

participants noted that their opinions did not change at all. One added, “Our deliberations were

mostly based on methodological questions, not big picture questions.”

A couple of respondents in the climate group answered that the ODC has potential as a

tool for graduate education on social and ethical issues arising from scientific research (Table

31). One respondent qualified the answer, “it needs a ton of work, but has a tone of potential.”


25

Similarly, other respondents pointed out specific problems, “[The ODC] is best used when it‟s

used often. However, I can‟t stand living in the meta-world of the internet from 9-5; and across

time zones. If the content were better limited, more progress and feelings of utility would result.”

Another respondent noted, “Asynchronous courses need to be properly set up.” Another

respondent expressed dissatisfaction, “I find that online interactions are difficult to sustain… I

would much rather interact with classmates in person, and feel that the course would have been

much stronger if we had more time in Montana to work on the wiki instead of sitting in lectures.”

As a follow up to Item 14 (Table 32), participants were asked whether the ODC has

potential as a tool for discussion between experts about complex social policy issues arising from

science and technology. Participants answered favorably overall, but with qualification. For

Example, “Sure, but…the discussions need to be structured and everyone needs to be speaking

the same language.” One respondent remarked, “The ODC certainly has promise, but cause must

be taken to direct the user experience better. Discussion pages are good for some things; but

much of the deliberation process should occur on the content pages...Rarely, perhaps never, did

opposing view points get fully described on the content pages.” And one respondent commented,

“Communication online will never be as strong as communication in person.”

Four respondents from the Climate change group elaborated on their response to Item 17

(Table 33), which gave the following prompt: “I felt comfortable expressing my opinions on the

ODC.” One respondent answered positively, stating, “People were interested and open to hearing

others‟ thoughts.” One respondent felt that it depended on the circumstance and offered, “Often I

felt no problem expressing my opinion about a subject. If my expertise were stretched, then I was

worried. It had more to do with the topic than the medium.” And a couple of participants

responded that they felt uncomfortable. For example, “I was not comfortable expressing my


26

opinions. For the most part I tried to stick to statements of fact.” Another participant felt that s/he

was unable to “fully understand certain contributions nor fully express my own.” S/he added,

“The human aspects of science, especially the ethical aspects, need to be addressed in a

humanistic manner.”

When asked for additional comments on improving the ODC, two participants responded

that they had no comment. One advocated for video conferencing. And one suggested, “Ability

to cut from ODC to word, and a strong connection between ODC and email. Also, attachments

etc. should open in separate windows!”

Biotechnology

Six participants from Biotechnology provided positive comments about the effectiveness

of the instructor (Item 6 Table 26). One respondent stated, “Dane did an excellent job with our

group, he worked very hard to keep us on track…He is very good at drawing conversations out

of participants and helping them develop and analyze their positions.” Another respondent

observed, “He was very good at encouraging our own opinions and not forcing his own.” And

yet another remarked, “The instructor did an excellent job posing important questions and

making sure that no participant‟s voice was drowned out.”

Eight participants from the biotechnology group responded to the question about whether

their position on the issues discussed in the course changed through their deliberations with the

other students. About half of the participants responded “yes” or “it did” and provided detailed

commentary. For example, “Yes; definitely. As I learned more and heard more perspectives of

my fellow deliberators I learned about newer aspects of ag biotech. To have a group of informed

and interested people to discuss a topic of this sort certainly necessitates learning more!” Several

of the respondents commented on the exchange among students. For example, “I learned a lot


27

from the other perspectives shared with my by other students.” One respondent discussed the

difference between consensus and deliberation:

Yes; sometimes even when we would agree on a particular response it was interesting

and profound to realize that we have different reasons for our respective responses. I

think this is important for deliberation; because it is intended to do more than come to a

consensus policy recommendation; but to reach a heightened level of understanding on

the issue. It can also be helpful to understand the motivation and rationale even when we

can‟t come to a true agreement.

Several respondents noted that their “position stayed the same,” but each gave a “nuanced”

response, maintaining, for instance, “I am much more interested in understanding the social

aspects of the GM debate,” or “I have examined and thought deeper about my positions.” One

respondent noted that although “My basic positions on biotechnology issues have remained more

or less the same…I certainly have a much more nuanced understanding and set of opinions about

more complex or grey areas.”

In elaborating on their response to Item 12 (Table 30), on the potential of the ODC for

graduate education, all eight respondents answered positivity. The strongly positive responses

ranged from observations about the important role that wiki software plays in our culture to the

benefits of using the ODC for graduate students. Benefits included, “greater level of

engagement,” and “having to write collaboratively.” One student claimed, “I think it

would…make students more informed citizens and aware of the main misunderstandings in

media and in society, and maybe even motivate them to make efforts to change misconceptions

in society.” Another respondent would have preferred face-to-face discussions, but also held that

“the wiki was a good alternative and an important one for an interdisciplinary group which is

spread across the country.” S/he went on to note some problems, “[S]ometimes there is so much

written in a short burst that very little true communication occurs as everyone is trying to submit

before a deadline or after a prolonged absence.” Similarly, another student noted that the ODC is


28

a “tool,” which “cannot form a course by itself, as interaction with fellow students and teachers

is an important aspect of learning.”

When asked to elaborate on Item 14 (Table 31) regarding the potential of the ODC for

discussion between experts about complex social policy issues, all but one of the seven

respondents answered positively. For example, a positive response, “In terms of a tool for

experts, I think one of the great aspects is the capacity to engage in an asynchronous discussion

that allows for geographically disparate people to participant on their own time schedule.”

However, some participants qualified their response. One respondent was concerned that experts

would use the ODC to “push their agenda” and suggested that “when experts participate in the

ODC the directors of the ODC should make sure they have a broad spectrum of experts with

varying positions to assist in deliberations…” Finally, one participant felt strongly that “for

experts who are discussing complex issues aimed at having an immediate impact on policy it is

not adequate. It could be a jumping off point, but it should be integrated into…conferences.”

When asked to elaborate about their comfort in expressing opinions on the ODC (Item

17, Table 33), the majority of the Biotech group felt comfortable. Of the five respondents, four

expressed that they were comfortable and noted that their comfort stemmed from getting to know

one another at the face-to-face workshop. For example, “It helped to know the other participants,

the weeklong workshop went a long way to breaking down communication barriers. Even when I

wrote something I thought someone else might disagree with, I could address their position and

even make a personal comment that I am...respectfully disagreeing with so-and-so, but here‟s

what I think.” Another participant remarked that [the ODC] leads to well constructed

deliberation and also accountability. One participant commented on the impersonal nature of the


29

wiki. “It makes it relatively easy to criticize other‟s positions and writing. It gives the impression

that communications are somewhat impersonal”

Suggestions for improving the technical aspects of the ODC included adding special

discussion pages where the “content itself can be discussed” like on Wikipedia, developing a

“tree diagram to picture how everything is linked together,” and adding a separate “track changes

page for each of the topics.” One participant suggested spending more time at the workshop

learning “the protocols and etiquette for working on this kind of a project.”

Nanotechnology

Five participants provided comments about the effectiveness of the instructor (Item 6,

Table 26). Most of the participants answered quite positively. The “instructor was an effective

organizer and guided the discussions. He was keeping track of everybody‟s work and pinpointed

difficulties and important questions.” And another noted, “He made sure that everyone‟s opinion

is being taken into consideration so that the articles [are] not biased.” One respondent claimed,

“Dr. Preston did a great job directing discussion and always asked if everyone had a clear idea of

what they would work on next.” One participant answered, “Since our group didn‟t really seem

to do much deliberating, there wasn‟t a whole lot the instructor could do to facilitate. He

encouraged and cajoled to set tasks. But there was little response in terms of discussion and any

real deliberation.”

When asked whether their position changed during deliberation with other students, the

five responses from the nanotechnology group were mixed. Two respondents replied that “yes”

their position had changed. One respondent noted that “my understanding of the central issues

that need to be addressed…has been greatly enhanced.” The other observed, “Initially I was very

against [regulation] thinking that it‟s all just empty fear…But…I have realized that ethics do


30

play a very important role in justifying things we discover.” One respondent noted that his/her

position changed “some,” and stated, “In certain cases I felt that communication was not very

good, since people didn‟t really understand each other‟s views.” One respondent reported feeling

“shut out,” because “engineers and scientists like to work differently than philosophers.” Finally,

one participant remarked, “it actually hardened my view in a negative direction.” This participant

held that there was “little deliberation/description” and that there was “a lack of awareness or

acceptance of significance.”

Four participants from the Nanotechnology group saw that the ODC has potential as a

tool for graduate education (Item 12, Table 30). However, a couple of participants qualified their

responses. “Though there were some serious issues that arose concerning the open-editing format

(angered colleagues etc), I think that the possibility for productive work is viable so long as the

group of writers share a common goal and a common methodological perspective.” Another

respondent thought it had potential, but focused on its technical issues including suggesting the

development of better threads and discussion pages.

When it came to the potential of the ODC as a tool for experts, two respondents noted its

potential, with qualification (Item 14, Table 31). For example, “The ease of informational access

that the wiki-page provides is an excellent source for academic debate and for the presentation of

all facets of a given subject. There are limitations…that have the potential for stunting growth.”

When asked if they felt comfortable expressing opinions in the ODC, responses were

mixed (Item 17, Table 33). One respondent noted, “I was comfortable with expressing debatable

viewpoints and was ready for discussion. Unfortunately, it did not always follow.” While another

respondent remarked that there was a “tension between the humanities and the sciences that kept

the free exchange of ideas from flowing freely.”


31

Two participants from the Nanotechnology group had suggestions for improving the

technical aspects of the ODC. One participant suggested “Making it easier to track changes for a

specific category.” The other participant suggested, “instead of having just one formal deadline,

the entire project should be divided into more stages consisting of deadlines for each one of

them.”

End of Semester Survey (Quantitative)

The following tables summarize the results from the end-of-semester survey. The

information presented here involves a range of issues. Several different questions involve student

evaluation of the Debating Science program as a whole, including the instructors, the face-to-

face workshop and the online semester course. Information gathered here also estimates the

extent to which participants were capable and comfortable of expressing thoughts and opinions

online. Students were also asked to evaluate the ODC in terms of its effectiveness for graduate

students etc. Finally, students were asked about the likelihood that they would engage in

interdisciplinary deliberations in the future.

Table 23

1. Please indicate the course in which you were enrolled.

Response Total Response Percent

Biotechnology 8 40%

Climate Change 6 30%

Nanotechnology 8 30%




32

Table 24

2. How many hours of work do you estimate you put into this course over the semester?


0 to 15 3 16%

16 to 30 3 16%

31 to 45 4 21%

46 to 60 4 21%

61 to 75 5 26%

Other, please specify 0 0%



Table 25

3. Overall I found the online course to be a useful educational experience

Strongly Agree Agree Neutral Disagree

Strongly

Disagree

Online 42.11% (8) 47.37% (9) 0% (0) 5.26% (1) 5.26% (1)



Table 26

6. The instructor was effective at facilitating the online deliberations


Strongly

Disagree

Instructor 52.63% (10) 31.58% (6) 10.53% (2) 5.26% (1) 0% (0)




33

Table 27

8. After taking this course I am better able to place the area of scientific research discussed in

the course into broader social and ethical contexts.


Strongly

Disagree

Context 42.11% (8) 47.37% (9) 5.26% (1) 5.26% (1) 0% (0)



Table 28

9. The interdisciplinary deliberations (interacting with the other students) expanded my

understanding of the issues discussed in the course.

Strongly


Strongly

Disagree

Deliberation 42.11% (8) 36.84% (7) 10.53% (2) 10.53% (2) 0% (0)



Table 29

11. The summer workshop prepared me well for the online portion of the course.


Strongly

Disagree

Workshop 42.11% (8) 36.84% (7) 15.79% (3) 5.26% (1) 0% (0)



Table 30

12. The ODC (deliberative-wiki format) has potential as a tool for graduate education on social

and ethical issues arising from scientific research.


Strongly

Disagree

Wiki 31.58% (6) 52.63% (10) 10.53% (2) 5.26% (1) 0% (0)



34

(skipped this

question) 4

Table 31

14. The ODC (deliberative-wiki format) has potential as a tool for discussion between experts

about complex social and policy issues arising from science and technology.

Strongly


Strongly

Disagree

Expert tool 36.84% (7) 42.11% (8) 10.53% (2) 10.53% (2) 0% (0)



Table 32

16. The ODC provides a forum for freely exchanging ideas and opinions.


Strongly

Disagree

Free exchange 31.58% (6) 63.16% (12) 0% (0) 5.26% (1) 0% (0)



Table 33

17. I felt comfortable expressing my opinions on the ODC.


Strongly

Disagree

Comfort level 47.37% (9) 31.58% (6) 10.53% (2) 10.53% (2) 0% (0)



Table 34

21. Do you agree that scientists need formal training in ethics?


Strongly Agree 13 68%

Agree 4 21%

Neutral 2 11%


35

Disagree 0 0%

Strongly Disagree 0 0%



Table 35

22. To what extent do you believe that it is good for scientists to participate in public

deliberations about directions for science and technology research?


Participation 52.63% (10) 47.37% (9) 0% (0) 0% (0) 0% (0)



Table 36

23. To what extent do you currently feel equipped to participate in ethical deliberation about

science policies?

To a great extent To some extent

To no extent at all

Preparation 47.37% (9) 52.63% (10) 0% (0)



Table 37

24. Do you anticipate being involved in any public policy deliberations in your chosen career?

Very Likely Likely Neutral Unlikely Very Unlikely

Expected involvement 42.11% (8) 52.63% (10) 5.26% (1) 0% (0) 0% (0)



Conclusions

Debating Science 2008 Survey Report

Participants in the 2008 Debating Science Program generally agreed about the need for

scientist involvement in public science debates. Similarly, participants agreed that the


36

effectiveness of public science debates were poor. Most of the participants anticipated being

involved in public debates about science, and felt that their training in the intellectual qualities of

deliberation was satisfactory or neutral.

Participants responded very positively to the summer workshop. The survey indicates that

the summer workshop was successful in raising student awareness about the ethical complexities

associated with emerging technologies and in increasing student confidence about participating

in interdisciplinary ethical dialogue. Participants indicated that they benefited from the lectures

as well as from interacting with peers, instructors, and experts. Suggestions for improvement

included making more time for discussion in lectures and group sessions, reconsidering the

content of the lectures, and changing the format. Overall, students benefited from the workshop,

and the majority rated it as excellent.

Reflections regarding the online portion of the program were considerably more varied.

Many participants reflected positively on the online chats and discussions, often citing the

benefits of interdisciplinary collaboration. However, participants also commented that the online

component was ambiguous, lacked focus, and required more structure. Also, several participants

remarked that their level of participation was affected by the format and group dynamics of the

online sessions. Many respondents remarked that it was difficult to edit each other‟s work online.

This report evaluated qualitative questions specifically pertaining to the ODC according

to group, since groups worked separately on the ODC. This revealed discrepancies between the

three groups regarding beliefs about the overall usefulness of the ODC, comfort levels on the

ODC, and the quality of deliberation on the ODC. Both the Nanotechnology group and the

Climate Change group reported mixed views on the ODC. Some respondents from these groups

had a positive experience deliberating on the ODC and felt that it had a lot of potential. Other


37

respondents listed several problems regarding the format and structure of the ODC, including

interpersonal dynamics, discomfort expressing opinions, and issues with the format. The

Biotechnology group, however, expressed generally positive remarks about the quality of

deliberation they experienced on the ODC and its potential as a deliberative tool. Participants in

the biotechnology group reported feeling comfortable expressing opinions on the ODC, and

editing each others work. Similarly, several members of the biotechnology group reported that

their positions had changed to some extent due to the online experience, while the

Nanotechnology and Climate Change groups reported more “hardened feelings.”

However, considering the responses of the participants to quantitative measures

collectively, reveals that the majority of the respondents agreed that the ODC provides a forum

for freely exchanging ideas and opinions and as a tool for discussing complex social and policy

issues arising from science and technology.

Comparison of Debating Science 2007 and 2008 Survey Reports

As a conclusion to the pilot course portion of the Debating Science program, the two

consecutive years of the program are compared. In keeping with the organization of the results

sections of both reports, the comparison of the two years will be compared according to each of

the three surveys. The following comments incorporate both qualitative and quantitative

elements.

In the pre-workshop survey respondents from both years of the program generally agreed

about the poor quality of public science debates. Participants felt that the public have poor

knowledge of scientific issues and for the most part emphasized the important role scientists

should play in these debates. Participants from both years of the program indicated that scientists

should receive training in ethics and expressed fairly neutral views about the quality of ethics


38

training received in science education. The majority of participants anticipated being involved in

public deliberation in their careers.

After the workshop, participants from both the 2007 and 2008 workshop commented

extensively on the quality of the lectures, the organization of the course, the benefit of peer-

interaction, and the quality of the instructors. At the same time, respondents from both years

offered suggestions for improving the lectures and the group work. Suggestions for improving

the lectures involved more time for discussion, for both years. Yet, the 2007 participants made

several comments regarding the quality of the ethics education received in the workshop and

several respondents suggested that the workshop could have been improved with more ethics

lectures. The 2008 participants, on the other hand, suggested improvements to the science

lectures and asked for more in depth or more introductory science lectures. A few respondents

from both years commented on the need for more training in deliberation and commented that

the workshop did not provide enough opportunities for practicing deliberation. Similarly, a

couple of participants from both years noted the lack of structure and the lack of obvious goals in

the group sessions. Several participants from both years commented positively on time

management. However, several participants from 2007 commented that the time management

was quite poor and felt that the workshop was overscheduled. Time management did not appear

to be an issue in 2008.

The end-of-semester survey for the 2007 and 2008 programs gathered information about

both the program as a whole and the online portion of the course. While surveys for both years

concentrated on evaluation of the actual course, the surveys had components that were

substantially distinct. The 2007 survey focusing much more on broader issues with respect to

perceptions of the ethics and science interface. The 2008 survey focused much more on


39

deliberation and the ODC. Thus several aspects of the end-of-the semester survey don‟t lend

themselves to comparison. Regardless, the surveys reveal several similarities and differences as

well as evidence of improvement.

Even though the 2007 survey respondents listed the ODC as one of the best aspects of the

course, participants seemed to benefit more from the face-to-face session. Several of the 2007

survey respondents mentioned technical, organizational and participatory issues with the ODC

throughout the survey and many 2007 respondents found the ODC difficult to navigate. The

2008 participants also noted organizational and participatory issues with the ODC, but responded

much more positively to it, listing it as among the best aspect of the program and noting the

collaborative aspects of the ODC much more.

There were several notable improvements in the 2008 program when compared to the

2007 survey. Both years mentioned the lack of goals and expectations of both the program as a

whole and of the ODC. Yet, this seemed to be a much more substantial and widespread

complaint in the 2007 report; the 2008 survey contained fewer participants who were dissatisfied

with the goals and organization. Participation improved notably in 2008. The majority of 2008

respondents reported spending over 30 hours on the course during the semester, whereas less

than half reported spending over 30 hours in 2007. In addition, the lack of student participation

was a concern that many voiced in 2007. However, fewer participants in 2008 mentioned this.

The logistics of the online portion of the course improved in 2008 as well. In 2007 participants

mentioned that switching online tools midway through was disruptive. In 2008 the online portion

seemed to go more smoothly, as such frustrations were not mentioned. The addition of scheduled

chat sessions was often positively mentioned in the report and may have contributed to the

notable improvement in 2008 participants‟ experience of the ODC.


40

Further, the 2007 end-of-semester survey included a qualitative question asking for

suggestions to improve the summer workshop for better preparing students for the semester

course. Two prominent suggestions included: defining goals, and training on the ODC. While a

few respondents mentioned the need for improvement in defining the goals of the course, the

majority of students in 2008 either agreed or strongly agreed (79%) that the workshop prepared

them for the online portion of the course. A few 2008 respondents suggested spending more time

on the ODC during the summer workshop, but for the most part participants seemed to feel better

prepared for the online course in 2008.

Suggestions for the Design of Future Surveys

We have a number of suggestions for future evaluation efforts of similar courses. One is

the keep the wording of similar items and response choices consistent across administrations of

different version of the survey as a means of comparing responses over time. Comparison of

groups over time should also be considered, however, it should be noted that a larger sample size

is necessary for this. Also, groups should be somewhat similar in basis for comparison. Also, we

recommend developing a more research instrument to more accurately measure participant

experiences during deliberation. All in all, however, the three surveys appeared to produce very

useful information.


41

Appendices

1) Pre-Workshop Survey

2) Post-Workshop Survey

3) End-of-Semester Survey


42

Appendix 1

Pre-Workshop Survey


43


44


45


46


47

Appendix 2

Post-Workshop Survey


48

Appendix 3

End-of-Semester Survey


49


50


51

report on 2008 debating science program survey science program/imx... · 2018-01-05 · debating...

Documents