Explaining and predicting outcomes of large classrooms using audience

response systems

Paul Benjamin Lowry

Information Systems Department, Marriott School Brigham Young University

Nicholas C. Rommano, Jr.

Department of Management Science and IS, Oklahoma State University

Randy Guthrie

California State Polytechnic University – Pomona

HICSS 2006 (IEEE)

1. Introduction

• A substantial base of evidence suggests that active learning techniques improve learning, achievement, and other related outcomes regardless of the subject being taught – compared to traditional lecture, e.g. (Gahr, 2003)

• However, supporting active learning in large classrooms is particularly difficult because interactions betweens students and instructors are more difficult and sometimes more intimidating. (Marbach & Sokolove, 2002)

• The focus of this paper is to develop ways to improve active learning through increased process interactivity in the traditional large classroom through what we term “interactive lean-media technology”. The technology we focus on is called audience response systems (ARS).

1. Introduction

• ARS are a form of computer-mediated communication (CMC) technologies that are used to improve participation and interaction in large classroom using a simple, lean-media interface, involving hand-held devices.

• These hand-held devices allow large groups of students to individually engage classroom content in real time while receiving instant feedback on their individual responses as well as the responses of all their classmates.

• ARS researchers have initially focused on its application to large educational groups, e.g. (Grthrie & Carlin, 1996). This line of research increasingly suggests great potential for using ARS to increase interactivity between students and instructors, e.g. (Dufresne et at, 2004). 中央大學陳德懷老師也有類似的研究

1. Introduction

• These studies are promising, however they leave many questions unanswered; because they lack any substantive theory to explain or predict interactivity in large ARS classrooms.

• This paper develops and explains theory that predicts how ARS can affect the level of process interactivity among learners in large classrooms and how the resulting increased interactivity should lead to improvements in communication quality, satisfaction, efficiency, and decreased status effects when compared to that of students in traditional large classrooms.

2. Theoretical model development

2.1 Interactivity

• The interactivity construct usually refers to two-way communication and interaction (Rafaeli, 1988) and research on this construct has generally focused on face-to-face(FtF) interpersonal communication (Burgoon et al, 2000).

• Groups that experience more reciprocity, control, and synchronicity will also experience more interactivity: Interactivity in groups is a positive function of reciprocity, control, and synchronicity.

2.2 Interactivity

• We now assert that increased interactivity in groups, through interactive media, will improve group performance.

• Hackman (1987) explained that the interaction process in work groups is critical to effectiveness because it can minimize loafing, promote a shared commitment among group members, foster sharing of expertise and collective learning, etc.

• Process losses in traditional large groups can be so severe that a large group can be rendered ineffective.

• Some of the process losses and dysfunctional behavior that increase with group size include evaluation apprehension, loafing, and production blocking (Steiner, 1972).

2.2 Interactivity

• Insight into how interactive media can promote interactivity in groups can be gained by further describing the capabilities of interactive media in light of existing research on interactivity, CMC and Group Support Systems (GSS).

• A substantial body of empirical and theoretical research on GSS shows that key features such as parallelism and anonymity help small groups overcome process losses and perform better than when there is no technology support (Dennis, 2001).

2.2.1 Parallelism and interactivity• Parallelism is the ability of group members to contribute infor

mation simultaneously (Dennis et al, 2001).

• Parallelism also results in reduced cognitive interference, because participants don’t have to wait to contribute their ideas (Dennis & Valacich, 1993).

• In summary, since parallelism increases the ability of group members to contribute information simultaneously, thereby reducing production blocking and cognitive inertia, parallelism provided through interactive media should increase the reciprocity, control, and synchronicity of large groups. (Hypothesis)

2.2.2 Parallelism and efficiency

• Since parallelism should increase interactivity in groups, which includes reciprocity, control, and synchronicity, time should be used more efficiently in terms of creating interactive discussion.

• Also, the reduction of production blocking and cognitive inertia allowed by parallel communication should also allow time to be used more efficiently. (Hypotheis)

2.2.3 Anonymity and status effects

• Anonymity often increases motivation of individual group members to participate (Dennis et at, 2001).

• Without anonymity, many participants – especially low status participants – may withhold ideas or comments due to evaluation apprehension (Dennis, 1996) and conformance pressure (Hackman & Kaplan, 1974). These negative effects due to status are also known as status effects (Weisband et al, 1995).

• As a result, anonymity should directly increase reciprocity and control in large groups and decrease status effects. (Hypothesis)

2.2.4 Communication quality

• A high quality group discussion develops the generation of multiple perspectives, where shared knowledge allows for understanding and the successful communication of problems solving (Burgoon, 2002).

• Interactive media are more likely to encourage multiple perspectives and shared knowledge than traditional oral communication.

• In group research, synergy is defined as the ability of a group to produce more and better ideas than would be produced by the sum of individual group members working alone (Dennis, 2002).

• In summary, communication quality is a positive function of interactivity and efficiency. (Hypothesis)

2.2.5 Satisfaction

• A recent theoretical model proposed by Khalifa and Liu (2003) states that overall satisfaction is a result of the gap between expectations, desires, and what a learner actually experiences.

• The expectation a learner has, and what he or she desires from the experience, will determine the overall satisfaction derived from that experience.

• In summary, satisfaction of group members is a positive function of interactivity, and communication quality. (Hypothesis)

2.3 Extension of theory to ARS

• Two foundational assumptions underlying this theoretical extension are that ARS is a form of CMC that has features compatible with key GSS interactivity features – namely, parallelism and anonymity.

• Based on the previous propositions, we hypothesize ARS as follows:

H1: Students in ARS classrooms will exhibit more perceived active control than in non-ARS classrooms.

Perceived controlPerceived group

interactivity

H1

• H2: Students in ARS classrooms will exhibit more perceived reciprocity than in non-ARS classrooms.

• H3: Students in ARS classrooms will exhibit more perceived synchronicity than in non-ARS classrooms.

• H4: Students in ARS classrooms will have higher level of efficiency than in non-ARS classrooms.

• H5: Students in ARS classrooms will exhibit fewer status effects than in non-ARS classrooms.

• H6: Students in ARS classrooms will exhibit higher perceived communication quality than in non-ARS classrooms.

• H7: Students in ARS classrooms will exhibit higher process satisfaction than in non-ARS classrooms.

• H8: Students in ARS classrooms will exhibit higher outcome satisfaction than in non-ARS classrooms.

3. Method

• The design of this study involves two conditions:

(1) large-class interaction and decision-making performed in classrooms using active learning techniques without ARS (control group);

(2) large-class interaction and decision-making in classrooms using active learning techniques in combination with ARS (treatment group).

Condition Mode Tools

(A) Control FtF No tools

(B) Treatment FtF + ARS ARS (CPS system)

3.1 Participants

• A total of 346 undergraduate business administration majors at a large, public Southern California university participated in this study.

• Participants’ gender was as follows: male (60.7%), female (39.3%). Average age was 22. Average GPA was 3.01.

3.2 Tool

• The ARS used for this research is the Classroom Performance System (CPS).

• The Classroom Performance System (CPS) provides a mechanism for teachers to electronically capture students' responses to questions and then immediately display the results through the classroom computer/projection system.

• The system consists of a receiver connected to the instructor's computer which records the responses given by students on small, hand-held units (so-called "clickers").

• The results are instantly available and can be immediately displayed to the class, saved on the professor's computer, or entered into the course records (e.g., WebCT, grade book, etc.).

3.1 Tool

3.3 Treatment and control procedures

• Both the treatment and the control classrooms had the same instructor, textbook, lecture presentation materials, course schedule, in-class assessments (quizzes), and project assignments.

• The course duration was 10 weeks.

• The instructor stood at ground level in front of the stage and presented written content and interactive questions via a laptop and LCD projector.

• Students in the treatment group had mastered the use of their response pads by this time and earlier concerns and problems that are a normal part of learning a new technology.

3.3.1 Treatment group

• The response pads were used for recording daily attendance, taking 5-point quizzes, and for answering questions asked during lectures and discussions.

• Students were given from 30-45 seconds to answer questions using their response pads.

• The CPS software was also used to facilitate discussions.

• During discussions, the instructor would ask a question soliciting experience or opinion information which then became the basis of discussion.

3.3.2 Control group

• Control students were asked the same questions at the same place in the lecture and discussions as the treatment group.

• The questions were projected visually using the same CPS software as the treatment group.

• Control group students responded by raising their hand when the answer choice was read by the instructor.

3.4 Measures

• We measured interactivity with a reliable instrument that measures three sub-constructs that are necessary for interactivity to occur: active control by participants (3 items, alpha=0.73), reciprocity (6 items, alpha=0.83), and synchronicity (5 items, alpha=0.82)

Note: All questions were responded to on a 7-point scale (strongly agree to strongly disagree)

3.4 Measures

• Communication qualityCommunication quality was measured with two well-developed measures, discussion quality from (Tan et al, 19999) (4 items, alpha=0.84) and task discussion effectiveness from (Burgoon, 2002) (8 items, alpha=0.80).

• SatisfactionSatisfaction had two measures. We used a validated self-reported process satisfaction measure derived from (Tan et al, 1999) (5 items, alpha=0.76), and outcome satisfaction measure developed by Souren et al. (2004) (6 items, alpha=0.93)

• Status effectsStatus effects were measured with an instrument from (Davison, 1997) (4 items, alpha=0.91).

• EfficiencyEfficiency was measured with an instrument from (Davison, 1997) (3 items, alpha=0.82).

4. Analysis and Results

5. Discussion

• Using ARS to perceive statistically significantly higher levels of interactivity in the classroom than non-ARS student – particularly in terms of reciprocity. (H1-H3)

• ARS students experienced more efficient and effective class time use than non-ARS students. (H4)

• ARS students experienced significantly fewer negative status effects in the classroom than non-ARS students. (H5)

• ARS students perceived higher communication quality in the classroom than the non-ARS students. (H6)

5.1 Contribution

• Several findings have clear appeal for improving education in large classrooms,

(1) it is beneficial to know that ARS can make better and efficient use of class time;

(2) several findings point to ARS improving active-learning components of large classrooms, including increased communication quality, and decreased negative status effects.

• Decreased status effects can help foster a more nurturing environment where students can feel more equal and less intimidated than in traditional large classroom that may have dominant students in them.

• This study would allow researchers to better understand the synergy that develops from combining ARS use and active-learning techniques.