Learning in the Field: Comparison of Desktop, Immersive Virtual Reality, and Actual Field Trips for Place-Based STEM Education

Jiayan Zhao* Peter LaFemina† Julia Carr‡ Pejman Sajjadi§ Jan Oliver Wallgrün¶ Alexander Klippelǁ

The Pennsylvania State University

ABSTRACT Field trips are a key component of learning in STEM disciplines such as geoscience to develop skills, integrate knowledge, and prepare students for lifelong learning. Given the reported success of technology-based learning and the prevalence of new forms of technology, especially with immersive virtual reality (iVR) entering the mainstream, virtual field trips (VFTs) are increasingly being considered as an effective form of teaching to either supplement or replace actual field trips (AFTs). However, little research has investigated the implications of VFTs in place-based STEM education, and empirical evidence is still limited about differences between students’ learning experiences and outcomes in VFTs experienced on desktop displays and field trips experienced in iVR. We report on a study that divided an introductory geoscience laboratory course into three groups with the first two groups experiencing a VFT either on desktop (dVFT) or in iVR (iVFT), while the third group went on an AFT. We compared subjective experiences (assessed via questionnaires) and objective learning outcomes for these groups. Our results suggest that, although students reported higher motivation and being more present in the iVFT group, they did not learn more compared to those in the dVFT group; both VFT groups yielded higher scores for learning experience and perceived learning outcomes than the actual field site visit. These findings demonstrate positive learning effects of VFTs relative to AFTs and provide evidence that geology VFTs need not be limited to iVR setups, which lead to considerable equipment costs and increased implementation complexity. Discussing the results, we reflect on the implications of our findings and point out future research directions.

Keywords: Virtual field trips, immersion, place-based education.

Index Terms: Applied computing—Education—Interactive learning environments; Human-centered computing—Visualization—Visualization design and evaluation methods

1 INTRODUCTION

1.1 Place-Based Learning in STEM Disciplines According to educators in earth sciences, field trips are a crucial, indispensable component of learning, for example, about geology,

to develop skills, integrate knowledge, and prepare students for lifelong learning [1, 2]. Field trips help bridge formal and informal learning [3], and students have the opportunity to study in the real world and learn from interacting with raw natural materials to reinforce classroom-based learning [4]. Knowledge and skills developed in earth sciences are understood to be best gained through active learning strategies where they can be learned and practiced through authentic learning activities [5].

However, learning in the field can present challenges due to issues such as safety concerns, including liability, and lack of accessibility for disabled students [6, 7]. Additionally, smaller universities and colleges may have limited resources. Lastly, geoscientists must use the outcrops that are available to them, which might not be the most ideal, or require weekend field trips to more distant outcrops. Even when instructors are able to overcome these logistical challenges, instructional shortcomings are often associated with field trips in the real-world settings. For example, during the field trip, instructors tend to lead students through a region until some specific geological phenomenon is discovered, stop to gather students around them, and talk about the phenomenon. Some students may not be able to gather around the instructor due to the crowd or rugged terrain. Furthermore, students with little or no prior experience in the field may have difficulty making detailed observations and taking meaningful notes about the phenomenon, thus missing the main points [3]. Additionally, the instructor has no control over environmental distractions such as inclement weather, lighting, and traffic noise. Any combination of these factors can result in a field trip experience that may be not accessible to all students or a suboptimal learning effectiveness.

1.2 Virtual Field Trips By definition, virtual field trips (VFTs) should comprise observation, interpretation, and skill practice using computer-generated representations of an actual field site [6]. The concept of providing field-based learning opportunities through virtual environments is not new [8, 9]. In the early 1990s, geology teachers at Duke University created 18 computerized field trips by integrating satellite images and topographic maps with 3D rock samples. These VFTs were placed on CD-ROMs for students in a geology introductory course to use as laboratory exercises [10, 11]. Since around 1995, the explosive growth of the internet and the World Wide Web has made it the dominant method of providing VFT experiences. Web-based VFTs experienced on desktop displays have been widely used within geology education for a variety of topics and education levels. For example, researchers at Arizona State University designed and developed a variety of VFTs that embedded diagrams, gigapixel images, and URLs to explanatory videos with 360° imagery of field sites [9]. Users can browse these multimedia resources, rotate their view in 360°, and navigate to new locations.

In 2016, immersive virtual reality (iVR) entered the mass market of gaming industry, following the mass diffusion of sophisticated iVR devices such as Oculus Rift, Sony PlayStation, and HTC Vive. An iVR game offers vivid and pleasant experiences to the player, and it has features that are deeply

*e-mail: juz64@psu.edu †e-mail: pcl11@psu.edu ‡e-mail: jcc77@psu.edu §e-mail: sfs5919@psu.edu ¶e-mail: juw30@psu.edu ǁe-mail: klippel@psu.edu

893

2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)

2642-5254/20/$31.00 ©2020 IEEEDOI 10.1109/VR46266.2020.00114

different from more conventional desktop-based experiences [12]. What strongly distinguishes iVR applications from traditional desktop ones is the level of immersion. Immersion is a description of a technology that can be applied to a broad range of paradigms [13, 14]. Slater and Wilbur [13] proposed that the level of immersion depends on the extent to which a system is capable of delivering an inclusive, extensive, vivid, surrounding, and matching illusion of reality to the senses of a human user. The last two aspects are found to be most related to the desktop-iVR distinction on immersion, and are briefly introduced in the following. First, surrounding indicates the extent to which the physical world is shut out, including field of view (FOV) and types of display. Miller and Bugnariu [14] indicate that a head-mounted display (HMD) or a surround projection should possess a much higher level of surrounding than a computer monitor presentation with limited FOV, while a large-screen projection with extended FOV is somewhere in between. Second, matching refers to whether there is a match between the user’s body-based cues and the information gathered on the displays. In other words, matching considers how well the technology approximates actions and movements in the virtual space [15]. For example, iVR systems use external tracking sensors to enable 3D tracking of the HMD and render the virtual world by obtaining the users’ head orientation and position in real time; that is, a highly immersive experience maps the user’s movements to the virtual space in a natural way, and the user’s body itself becomes the main interface for interacting with the virtual world [12]. In contrast, desktop users typically use more abstract navigation interfaces (e.g., keyboard, mouse, or joystick) to control their ability to look around, meaning that desktop users “pan” through the virtual space to change their viewing direction without moving their bodies or turning their heads.

Consequently, iVR systems may alter the emotional experiences of users by significantly enhancing their sense of being in the virtual world. Presence, defined as “the sensation of being physically situated within a spatial environment portrayed by the medium” [16, p. 156], is of critical importance in relation to the development of enjoyable iVR experiences. It has been widely demonstrated that, compared to the traditional desktop application, the increased immersion delivered through an HMD can provide a stronger sense of presence [17–20]; in fact, it is expected to be highly engaging and absorbing. Thanks to its unique characteristics in providing an embodied, authentic learning environment, the emergence of iVR technologies leads to incomparable opportunities for researchers to advance virtual field trips (VFTs) to an unprecedented height (e.g., [7, 15]). However, it is important to note that, despite the highly fascinating and engaging experience, learning about geology in iVR is still an understudied phenomenon and to our knowledge, empirical studies have not convincingly answered the question of whether using iVR to teach about science would lead to better learning outcomes.

1.3 Immersion in Learning A highly immersive experience should lead to positive learning effects for many reasons [15, 21]. From a technical perspective, iVR is characterized by the use of an HMD, enhanced with haptic, auditory, and other sensory feedback to afford embodied experiences1 by mimicking natural embodied interactions through a user’s egocentric perspective [16]. Therefore, the virtual world is perceived as unmediated and users often process information of stimuli in the virtual world as they would from the outside world

1 Embodied experience refers to experiencing our self as being inside a body that feels “ours” by integrating the different sensory signals arriving to our body [22, 23].

[15, 24]. From this perspective, high-immersion iVR has the potential of scaffolding situated learning by integrating authentic contexts and situated activities into a tacit and unstructured learning process. Situated learning can increase the speed of knowledge gain because the simulation of real-world context for users to actively engage with supports near transfer of acquired knowledge and skills to their future learning [21, 25]. This idea is rooted in embodied cognition research, which states that the representation of knowledge is grounded in a person’s experience of perceiving and interacting with the environment. In other words, the visual, auditory, verbal, and kinesthetic aspects of an individual's experience are deeply integrated with the system of knowledge representation [26]. Therefore, experiencing an immersive virtual environment from an embodied, egocentric perspective may lead to deeper cognitive processing of the learning material, which can enhance knowledge understanding and transfer.

There is research supporting the educational value of iVR simulations (e.g., [7, 15, 27–29, 29]). One study by Markowitz et al. [15] investigated the effect of iVR experience on learning about marine science. In this study, participants experienced an immersive underwater world designed to show the process and effects of rising sea water acidity. Their knowledge of ocean acidification and attitude toward climate change were assessed before and after the iVR intervention. After an iVR experience, participants reported positive knowledge gain and an interest in knowing more about the causes and effects of ocean acidification. However, the researchers did not include a control group in their study, so the increase in assessment scores cannot be attributed solely to the use of iVR as it may be due to other factors such as repeated exposure to the same questions. In a more recent study, Klippel et al. [7] compared geology learning in a physical field site versus in an immersive virtual environment. Students in an introductory geoscience course were split into two groups with one group attending a traditional actual field trip (AFT), while the second group experienced the same virtually using an HMD. Compared to the AFT, this immersive VFT led to increased enjoyment, learning experience, and actual lab scores.

Some studies have shown a reversed effect of iVR on science learning. For example, Makransky et al. [19] compared learning from a science simulation via a desktop display versus an HMD, showing that participants felt a greater sense of presence when using the HMD to explore a virtual science laboratory, but they actually learned less compared to those experiencing the same lab simulation on a desktop display. Similar reduced learning effects of iVR compared to desktop has also been found in a more recent empirical study of learning life science in classrooms [30]. In this study, college students learned about how the human body works either using an interactive biology simulation with iVR or via self-directed PowerPoint slideshow on a traditional desktop computer. In the testing phase, students completed a post-test consisting of factual and conceptual questions about mechanisms of the human body. Despite the higher interest, motivation, and engagement on the material in the lesson, students in the iVR group performed significantly worse as measured by grades in the post-test than those in the desktop slideshow group. Makransky et al. [19] suggest that immersion may not be positively correlated with learners’ performance, presumably due to the extra cognitive load imposed by the iVR system. According to cognitive load theory, seductive details (i.e., interesting but irrelevant material) in the immersive learning environment can create extraneous cognitive load (i.e., the cognitive processing that does not support the learning objective; see [31] for a review) and hence lower learners’ cognitive interest [26]. Specifically, iVR systems that induce a high level of presence may interfere with reflection during learning, especially in the situation in which added

894

immersion is not directly relevant to the instructional goal; thus the added perceptual realism when using iVR systems could be categorized as seductive details that distract learners from the essential content [30]. Finally, similar levels of knowledge gain have been found between desktop and iVR in other learning topics (e.g., wayfinding and navigation: [32–34]; professional training: [20, 35, 36]).

Due to the reported mixed effects of iVR on science education, new research is needed to investigate deeply the role of immersion that comes into play when students learn about geology through experiences in a VFT. To our knowledge, so far the majority of empirical studies focusing on geology VFTs have primarily used desktop displays as the medium, and have suggested a supplementary or alternative role of the VFT for its physical counterpart in field learning [3, 6, 9, 37, 38]. These studies, however, are constrained by the limitations of technologies and hence neglect media effects for geology learning. With a rapid progress in the technical level of HMDs that has been observed in the past few years, adopting advanced and commercial HMDs allows us not only to probe media effects of VFTs in learning about geology but also to re-examine the educational value of VFTs added to AFTs.

In an attempt to address the particular aforementioned aspects of geology learning we present an empirical study as part of two sections of an introductory geoscience laboratory course conducted in Fall 2019. We investigated whether iVR was an effective medium to generate positive learning experiences and outcomes compared to a conventional desktop learning practice. Furthermore, we investigated the effectiveness of VFTs by contrasting directly the virtual with the actual field trip with respect to students’ learning experience and perceived learning outcomes. In the next sections we give an overview of the VFT design and describe an experiment to: 1) measure the VR features, system usability, sense of presence, and in-VFT testing performance of the same geology field trip experienced in iVR and on the desktop display, and 2) compare students’ learning experience and perceived learning outcomes after a traditional actual field trip (AFT), after a desktop VFT (dVFT), and after an immersive VFT (iVFT). Discussing the obtained results, we detail our efforts to systematically address the effectiveness of VFTs in geology education and suggest some directions for future work.

2 METHOD

2.1 Salona Field Trip Experience To evaluate the value of VFTs and establish a baseline, we selected an introductory geoscience course lab named Sedimentary Rocks: The Salona Formations (at an anonymous university). The Salona lab included a site visit to the Salona Formation along the east side of Highway 322 By-Pass, PA. This formation represents the lower part of the Upper Ordovician period, about 458 to 450 Ma. This formation shows the transition from shallow, subtidal carbonate rocks from the passive Laurentian margin to siliciclastic rocks sourced from the Taconic volcanic arc. The section has interspersed volcanic ashes (bentonites) that provide absolute age constraints on these transitions. The goals of this lab are to: 1) describe the lithologies that make up the Salona Formation; 2) study their arrangement stratigraphically and reconstruct the depositional environment through time; and 3) understand the geodynamic history of this outcrop in the broader context of the regional geology and plate tectonic paradigm. Students participated in small-sized groups of two or three out of a lab size of 25. Two TAs guided student groups through the field site experience with two stops; students had access to additional information through their lab manuals and class notes. At Stop 1, students were given an overview of the

Salona Formation and listened to the TA describing the outcrop2. At Stop 2, students walked along and observed the outcrop from a close distance, during which they made detailed observations, took notes, and accumulated first-hand data in order to complete individual laboratory assignments. The field site visit lasted 30–45 minutes.

The virtual versions were created with Unity3D [39] and closely mimicked the AFT including the order of the site visit, delivered teaching content, and additional information that students had access to during the field trip. Figure 1 is a map with locations that illustrates the procedure of the VFT experience. A total of 12 discrete locations were rendered by 360° images taken with a high-resolution (108K) Panono camera for the general site exploration. Note that we took 360° images not only at normal eye level but also at the height of 27 feet using a rather long tripod, a MegaMastTM. When exploring each location, students were offered access to an elevated perspective in addition to the normal ground perspective. Our earlier studies have suggested that such elevated views are often of critical importance to understanding a complex environment [16, 40]. Considering the selection of 360° imagery for simulating the field site, this design choice was consistent with a series of VFTs that have been developed for place-based STEM disciplines (e.g., biology [41], geosciences [7, 42, 43], and energy [44]). Compared to a fully 3D virtual environment, 360° images provide a more realistic visual representation of the field site and can be easily integrated with other sources of information into different VR platforms. The latter aspect is especially important as the Salona VFT was developed to facilitate introductory geoscience education. VR headsets that students have easy access to, such as Samsung Gear VR and Oculus Go, are relatively low in price but often come with limited memory capacities and graphics capabilities, not enough to support experiences in fully interactive virtual environments with authentic 3D objects.

Figure 1: Satellite image of the actual study area. Circles represent

locations where 360° images were taken and are numbered in order in which students were guided through them in the VFT. Four locations (Stops 1–4) had audio guidance and access points to additional information. Location 5/Stop 3 was the entry to a 3D model of the outcrop for measuring the stratigraphy (Source: Google Earth).

When viewing a 360° image taken at the normal ground perspective, students were able to navigate to other locations by selecting arrows on the ground (Figure 2, left). Similar to the AFT, students in the VFT started at Stop 1 where they observed the Salona Formation from a distance. Students were then instructed to teleport to Stop 2 and moved along the road cut through 360° images taken closely to the outcrop (including Stops

2 Note that not all AFT participants visited Stop 1, because students from one section of the course skipped this stop and directly started from Stop 2 during the field site visit.

895

3 and 4). At each stop, the 360° image contains 2–5 access points with additional information. Specifically, students could select information icons stitched onto the 360° image to access supplementary information from the lab manual (e.g., regional contour map of the Salona Formation) or higher-resolution photos mimicking hands-on learning experiences in the actual field site (e.g., hand specimen of sedimentary rocks that have abundant, intact fossils; see Figure 3). The 360° image along with each piece of additional information was associated with a short audio narration that was pre-recorded by the course instructor. Students heard an audio tour that either guided them through ongoing observations and assessments or that explained specific features of the Salona Formation, such as the stratigraphy visible in the 360° image or fine-grained carbonate muds deposited in shallow water. At the end of each stop (1–3), students were instructed to answer two questions that were related to their field observations and interpretations, or factual knowledge of the outcrop that they learned from pre-field trip in-class lectures, the lab book and audio narrations (Figure 2, right).

Figure 2: Left: Arrow on the ground allowed for navigation from one

location to the next. Right: Students were tested on their knowledge about the Salona Formation and geologic principles during the VFT.

Figure 3: Selecting the information icon stitched onto the 360°

image (left), additional information derived from the lab manual and high-resolution photos appeared (right).

After finishing the audio tour in the last 360° image (Stop 4), students retraced the road cut in reverse to Stop 3. At this location, students were transported to a 3D photorealistic model of the Salona Formation (surveyed using a handheld DSLR camera and built with structure from motion photogrammetry in Agisoft Photoscan [45]) for a strata measurement task (Figure 4, top left). Students were guided by a humanoid robot to measure the thickness of rock layers along a 1.5 m section of the outcrop (Figure 4, bottom). A data board, which displayed the set of measured lengths, allowed students to review and edit the collected data (Figure 4, top right). The measurement results along with a screenshot of the outcrop model were sent to students afterward so that they could make a detailed stratigraphic column including information about bed thickness, sedimentary structures, and color (as part of the lab assignment—required for both AFT and VFT). The whole virtual experience was approximately 20 minutes long.

Leveraging cross-platform support of the Unity game engine, the Salona VFT was developed both as a desktop and an iVR version with the same content and features. In the desktop VFT (dVFT), students sat in front of a desktop screen, and pressed the

WASD keys to look around and the left mouse button to interact with the virtual content (e.g., either selecting arrows on the ground to navigate or clicking highlighted icons for additional information). In the immersive VFT (iVFT), the experience was the same except the students wore an HTC Vive Pro HMD, stood in the center of the tracking area, and used one Vive controller to interact with the environment. The strata measurement task was designed differently in dVFT and iVFT. In contrast to our iVR version where a student could walk around to operate a virtual ruler attached to his/her hand controller in six degrees of freedom (Figure 4, bottom left), students in the desktop version were placed in front of the outcrop model. To generate length data, dVFT students hovered the mouse cursor over the outcrop model and pressed the left mouse button to draw nodes on the rock surface. Each pair of nodes was connected by a straight line segment for length computation (Figure 4, bottom right).

Figure 4: Strata measurement activity. Top left shows the 3D

photorealistic model created for a portion of the Salona Formation with an indication of which part students were to measure. Bottom: The robot teacher guides the student to perform the measurement either with the ruler tool used on top of an HTC Vive controller (iVFT group; bottom left) or using the mouse (dVFT group; bottom right). Top right shows the data board where measurements were recorded and deleted.

2.2 Participants and Design Fifty-two undergraduate students were recruited from two sections of an introductory geoscience laboratory course to participate in the virtual field trip (VFT) before going on the actual field trip (AFT), or consented to participate as a control group while taking part in the AFT only (26 AFT participants, average age 19.5 years, 11 females). The course was required for geoscience and related majors and engineering students, and was dominated by the latter. All participants were entered into a raffle for six $50 ice cream vouchers. Participants who agreed to attend the VFT were randomly assigned to the desktop VFT (dVFT) group (13 participants, average age 19.4 years, 5 females) and to the immersive VFT (iVFT) group (13 participants, average age 18.9 years, 7 females). The VFT participants received $10 for the time they spent in addition to the AFT.

The study employed a between-subjects design to compare the perceived VR features, usability, sense of presence, learning experience, and in-VFT test scores in the two VFT groups. Specifically, we compared:

dVFT group: Individuals were seated at a desk on which a MSI gaming laptop running an NVIDIA GoForce GTX 1080 graphics card with 17.3" Full HD

896

(1920x1080) was positioned. Participants wore headphones and used the mouse and keyboard to experience a 20-minute VFT to the Salona Formation.

iVFT group: Participants stood in the center of the tracking area, wore an HTC Vive Pro HMD with a display resolution of 2880 x 1600 pixels and a field of view (FOV) of 110°, and held one hand controller to experience the iVFT. The field trip experience was rendered by an iBuyPower computer equipped with a GeForce RTX 2080 Ti graphics card.

To examine the educational value of these two types of VFTs against a traditional field trip experience, this study also included a control group in which students participated in the AFT only. Note that students who had experienced the VFT also went on the AFT about one week after the VR experience. Data from their AFT experiences are not reported in this article.

2.3 Procedure and Measures The experimental procedure was different for the VFT and AFT groups, which is briefly described below.

Sign-up for the VFT experience was organized through a web-based calendar and up to two students could participate simultaneously in a one-hour experimental session at the authors’ home institution. Upon arrival, participants provided basic demographic information (e.g., gender, age, and ethnicity) and their attitude and opinions toward VFT experiences, and completed a self-report measure of technology enjoyment through Google Forms. Participants were given training to familiarize them with the basic functions and interactions of the VFT and then went through the virtual experience. After completing the VFT to the Salona Formation, participants answered the set of self-report questionnaires including VFT attitude, simulator sickness, VR features, system usability, sense of presence, learning experience, and perceived learning outcomes. Students in the control group did not experience the field site virtually but instead participated in the AFT only. They received a post-questionnaire via email and filled it out online. Students were asked demographic and self-report questions concerning spatial situation model, learning experience, and perceived learning outcomes. Lastly, all participants were asked to complete an official lab assignment through the general class assignment system. Table 1 presents an overview of the measures we used to assess the students’ experiences.

3 RESULTS As a first step in our analysis, we compared the three groups (AFT, dVFT, and iVFT) based on their self-assessed enjoyment of technology. There was no significant difference, F(2, 49) = .82, p = .45, η2 = .03. This result, while limited, indicates the three groups do not show any biases toward virtual experiences.

3.1 Desktop Versus Immersive Virtual Field Trips Independent two-sample t-tests were set to evaluate the media effects of geology learning by comparing dVFT with iVFT groups (see Table 2). The assumptions of normality and homogeneity of variance were met. There was a significant group difference for simulator sickness. Although the value was very small, iVFT participants reported significantly higher levels of simulator sickness than dVFT participants after experiencing the VFT. There was also a significant group effect on the sense of presence, such that the self-location score of iVFT participants was significantly higher than those in the dVFT group. Bayesian t-tests were conducted to boost statistical power for non-significant results presented in Table 2, and therefore produced Bayes Factors using the Bayesian information criteria. All the estimated Bayes

Factors [.38, .57] are larger than .33 but smaller than 1, indicating anecdotal evidence in favor of the null hypothesis (H0; i.e., the values of the target variable were similar between the two VFT groups; according to [47]). iVFT and dVFT participants seemed to yield similar scores for VR features, usability, and in-VFT tests.

Table 1: Overview of variables and instruments. Variable Description Assessment Individual differences: - Demographics - Technology enjoyment

Individual differences to discern participants, which provide insights into the group comparability.

Two technology enjoyment questions: 1) I enjoy using technology. 2) I enjoy playing video games.

Opinion on the virtual field trip (VFT)

Using a pre- post-design allows for assessing pre- to post-lesson changes in students’ opinions and attitudes toward VFTs.

Combination of five questions. Example questions: 1) I would like to see more use of virtual field trips in university teaching. 2) Virtual field trips can replace actual field trips.

Simulator sickness Simulator sickness (e.g., nausea, disorientation) is typically experienced by users during VR exposure. Severe simulator sickness might draw users’ attention from the VFT, decrease their engagement in the virtual field site, and hence reduce their learning.

The standard Simulator Sickness Questionnaire (SSQ [48]) – a self-report symptom checklist assessing 16 symptoms that are associated with simulator sickness.

VR features: - Representational fidelity - Immediacy of control

Technology features including the realism factor, degree of realism of the objects portrayed in the virtual environment; and the control factor, the amount of control the user had on activities in the virtual field site [49].

Combination of two questions for each technology factor that is presumed to have an indirect effect on the learning outcomes (modified from [49]).

Usability: - Perceived usefulness - Perceived ease of use

The evaluation of usability in VR systems is important for identifying and subsequently improving the user interface. It is also necessary to recognize the system features that were more or less essential for a VFT experiences.

The usability of a VR system was measured according to the attitude and perception of its users. The key usability factors of VR systems including usefulness (e.g., “valuable”) and ease of use (e.g., “unburdensome”) were evaluated in the current study.

Presence: - Spatial situation model - Self-location

Media effects researchers have curated substantial questionnaires to address questions of how present someone feels during exposure to a mediated location. We were particularly interested in a comparison of virtual field trip using the desktop computer and the HTC Vive.

We selected a subset of questions from a well-cited presence questionnaire (MECSPQ [50]) that measured the spatial situation model and self-location, as two levels of developing spatial presence through experiences in the virtual field site (i.e., two-level model [51]).

Learning experience: - Motivation: Field trip enjoyment - Cognitive benefits - Control & active learning - Reflective thinking

The four internal psychological factors are essential aspects of a learning experience. Their measures can provide evidence of what kind of learning experience is enhanced by the field trip and how important the learning experience is in shaping the learning outcomes.

Learning experience was individually measured by a combination of two questions for each psychological factor (modified from [49]), which take central stage in affecting the learning outcomes in the field trip.

Learning outcomes: - Perceived learning effectiveness - Field trip satisfaction - In-VFT test scores

A central purpose of learning is to acquire knowledge and increase the capacity to take effective action. Research suggests that field trips can improve students’ achievement, their attitudes toward learning, and their evaluation of the learning experience [49].

Learning outcomes focus on two domains, that is, the self-assessed affective domain in terms of perceived learning effectiveness and satisfaction with the field trip experience, and the performance achievement (i.e., correct answers count of questions asked during the VFT).

Note. In-VFT test scores were coded 0-6. Other variables were scored on a 5-point Likert scale or calculated by averaging the scores of several questions.

Concerning participants’ opinions on VFTs, a generalized linear mixed model was conducted to probe whether participants changed their general attitude toward VFTs after the VR intervention. Specifically, VFT group (desktop vs. immersive) and time (pre vs. post) were modeled as fixed effects of the VFT attitude, and participants were modeled as a random effect. The main effects of the VFT group, χ2(2) < .001, p = 1.0, or the time, χ2(2) = 1.53, p = .22, were not statistically significant. There was a significant interaction effect of the VFT group and time on VFT attitude, χ2(2) = 4.79, p = .03. However, pairwise comparisons with Tukey corrections using the emmeans package in R [52] did not show any significant results, t.ratio(24)’s > -2.5, p’s > .08. To test whether there was evidence in favor of H0 (i.e., there was no effect of VFT attitude over time or between VFT groups), we examined the data by conducting two Bayesian paired t-tests. The first paired t-test compared the pre-attitude of dVFT participants (M = 3.29, SD = .58) to their post-attitude (M = 3.2, SD = .63), BF10 = .57, which indicates anecdotal evidence in favor of H0. The second paired t-test compared pre- with post-attitudes for iVFT participants (iVFT pre-attitude: M = 3.06, SD = .48; iVFT post-attitude: M = 3.43, SD = .69), BF10 = 1.07, which indicates anecdotal evidence in favor of the alternative hypothesis (H1). iVFT participants seemed to have more positive attitude toward VFTs after experiencing the VFT while such difference was not found in the dVFT group.

897

Table 2: Mean comparison on simulator sickness, VR features, usability, self-location, and in-VFT test scores.

Variables M SD 95% CI t (df) p d BF10

Simulator sickness dVFT 1.15 .20 [-.44, -.08] -3.03 (22.93) .006** -1.19

iVFT 1.42 .25 Representati-onal fidelity dVFT 4.0 .84 [-.69, .46] -.41 (20.58) .68 -.16 .39

iVFT 4.12 .55 Immediacy of control dVFT 4.38 .82 [-.47, .85] .6 (23.99) .55 .24 .42

iVFT 4.19 .80 Perceived usefulness dVFT 4.15 .75 [-.89, .26] -1.11 (23.65) .28 -.44 .57

iVFT 4.46 .66 Perceived ease of use dVFT 4.23 .63 [-.71, .48] -.40 (22.48) .69 -.16 .38

iVFT 4.35 .83 Self-location dVFT 2.99 1.07 [-1.62 , -.11] -2.40 (21.44) .026* -.94

iVFT 3.86 .75

In-VFT test scores

dVFT 4.92 .86 [-.46, .93] .69 (24.0) .50 .27 .43

iVFT 4.69 .85

Note. CI = confidence interval, Cohen's d = effect size, BF10 = Bayes Factor supporting the alternative hypothesis over the null hypothesis.

Concerning participants’ opinions on VFTs, a generalized linear mixed model was conducted to probe whether participants changed their general attitude toward VFTs after the VR intervention. Specifically, VFT group (desktop vs. immersive) and time (pre vs. post) were modeled as fixed effects of the VFT attitude, and participants were modeled as a random effect. The main effects of the VFT group, χ2(2) < .001, p = 1.0, or the time, χ2(2) = 1.53, p = .22, were not statistically significant. There was a significant interaction effect of the VFT group and time on VFT attitude, χ2(2) = 4.79, p = .03. However, pairwise comparisons with Tukey corrections using the emmeans package in R [52] did not show any significant results, t.ratio(24)’s > -2.5, p’s > .08. To test whether there was evidence in favor of H0 (i.e., there was no effect of VFT attitude over time or between VFT groups), we examined the data by conducting two Bayesian paired t-tests. The first paired t-test compared the pre-attitude of dVFT participants (M = 3.29, SD = .58) to their post-attitude (M = 3.2, SD = .63), BF10 = .57, which indicates anecdotal evidence in favor of H0. The second paired t-test compared pre- with post-attitudes for iVFT participants (iVFT pre-attitude: M = 3.06, SD = .48; iVFT post-attitude: M = 3.43, SD = .69), BF10 = 1.07, which indicates anecdotal evidence in favor of the alternative hypothesis (H1). iVFT participants seemed to have more positive attitude toward VFTs after experiencing the VFT while such difference was not found in the dVFT group.

3.2 Virtual Versus Actual Field Trips To examine the learning effectiveness of VFTs in contrast to an actual field trip (AFT) experience, we examined the main effects of AFT, dVFT, and iVFT on the spatial situation model, learning experience, and perceived learning outcomes, using the one-way analyses of variance (ANOVAs) (see Table 3). The assumption of homogeneity of variance was met. There were significant differences for field trip enjoyment. In the Tukey HSD post-hoc test, AFT participants (M = 3.37, SD = .96) reported significantly lower field trip enjoyment than those in the two VFT groups (dVFT: M = 4.08, SD = .73; AFT-dVFT: 95% confidence interval [CI] = [-1.38, -.04], p = .04, d = -.29; iVFT: M = 4.62, SD = .51; AFT-iVFT: 95% CI = [-1.92, -.58], p < .001, d = -.46). Although there was no significant difference comparing field trip enjoyment

between dVFT and iVFT groups (95% CI = [-1.31, .24], p = .22, d = -.16), a BF10 of 1.93 indicates anecdotal evidence for H1, suggesting that iVFT participants seemed to enjoy their field trip experiences more than dVFT participants. There were also significant differences for cognitive benefits (referring to easier comprehension and a better overview of the content learned). The Tukey HSD test revealed an overall advantage of both VFT groups (dVFT: M = 4.27, SD = .81; iVFT: M = 4.12, SD = .71) compared to the AFT group (M = 3.31, SD = 1.01) (dVFT-AFT: 95% CI = [.23, 1.7], p = .008, d = .37; iVFT-AFT: 95% CI = [.07, 1.54], p = .03, d = -.32). The difference between the two VFT groups was not statistically significant, 95% CI = [-.7, 1.0], p = .9, d = .05, BF10 = .4. Similar effects were observed in the comparison across all three groups for field trip satisfaction. Specifically, both dVFT (M = 4.35, SD = .83) and iVFT (M = 4.27, SD = .78) participants reported significantly higher field trip satisfaction than AFT participants (M = 3.31, SD = .91) (dVFT-AFT: 95% CI = [.33, 1.74], p = .002, d = .4; iVFT-AFT: 95% CI = [.26, 1.67], p = .005, d = -.38), whereas the two VFT groups seemed to yield similar field trip satisfaction, 95% CI = [-.74, .89], p = .97, d = .03, BF10 = .37.

Table 3: Summary of spatial situation model, learning experience, and perceived learning outcomes in relation to differences

between actual field trip, desktop virtual field trip, and immersive virtual field trip groups.

Variables F (df1, df2) p η2 BF10

Spatial situation model 1.60 (2, 49) .21 .061 .51

Field trip enjoyment 10.76 (2, 49) < .001*** .31

Cognitive benefits 6.42 (2, 49) .003** .21

Control and active learning 1.88 (2, 49) .16 .071 .61

Reflective thinking 1.79 (2, 49) .18 .068 .58

Perceived learning effectiveness 2.64 (2, 49) .08 .097 1.08

Field trip satisfaction 8.86 (2, 49) < .001*** .27

Note. η2 = effect size, BF10 = Bayes Factor supporting the alternative hypothesis over the null hypothesis.

For non-significant results obtained from one-way ANOVAs, we followed up with Bayesian ANOVAs to examine if there was evidence for H0 (see Table 3). The estimated Bayes factors for spatial situation model, control and active learning, and reflective thinking indicate anecdotal evidence in favor of H0. AFT, dVFT, and iVFT groups seemed to yield similar scores for these self-report measures. For perceived learning effectiveness, a Bayes factor of 1.08 indicates anecdotal evidence in favor of H1. Given that some effect might exist across field trip groups, multiple comparisons with Bayes factors were used to test equality constraints. Because we wished to probe whether VFTs have any educational value added to the AFT, we combined the dVFT with the iVFT group and then compared the merged VFT group with the AFT group on perceived learning effectiveness, BF10 = 2.36. According to Bayesian principles, H1 was preferred, indicating that AFT participants seemed to have lower perceived learning effectiveness than those in the two VFT groups. Next, we tested if there was a difference between dVFT and iVFT by calculating the ratio of BF10 for the Bayesian ANOVA and BF10 for the equality constraint. The null hypothesis was preferred by a factor of .46, suggesting that the two VFT groups seemed to yield similar perceived learning effectiveness.

4 DISCUSSION Participants in the iVFT group reported significantly higher self-location scores and greater levels of simulator sickness than those

898

in the dVFT group. Here, we identify two VR features that may be related to the results: VR locomotion mode and display field of view (DFOV). A brief discussion of each feature is given below.

First, the VR locomotion mode affects users’ perception of motion cues. When participants experienced the dVFT, visual changes on the desktop display were initiated from button-pressing or mouse-clicking, which were independent of motion cues associated with bodily actions (e.g., walking, head rotation). In the iVFT, the HTC Vive Pro was equipped with positional tracking sensors, which afforded participants an egocentric, embodied access to the virtual field site. According to the embodied cognition framework [53, 54], a sense of presence is experienced when body-specific cues, such as vestibular and kinesthetic feedback, are perceptually grounded, rather than abstractly approached in the virtual environment. Compared to dVFT participants controlling visual changes through abstract navigation interfaces (i.e., mouse and keyboard), iVFT participants had access to their surroundings by using the same embodied activities used to look around in the real world, which could result in a greater sense of presence, or self-location. On the other hand, such embodied, egocentric access to an environment may be associated with simulator sickness in some cases. For example, we used 360° images taken at 27 feet above ground to provide a better overview of the field site; however, not everyone could benefit from this elevated perspective. If a user had acrophobia, the embodied experience afforded by iVR was likely to make his or her symptoms worse. Besides, display update lag or time delay in the Vive Pro HMD might have resulted in a sensory conflict between the visual and vestibular systems [55]. Simulator sickness was likely to occur due to such incongruent sensory input regarding motion between the head movement and visual changes while in the iVFT. In contrast, position trackers were not used with the desktop display when dVFT participants experienced the field trip, meaning that the dVFT could result in less sensory conflict and hence induced less simulator sickness than the iVFT.

Second, display field of view (DFOV) is the FOV permitted by the physical dimensions of the display [56]. The HTC Vive Pro HMD afforded a much wider DFOV than the 17.3’’ desktop display used in the current study. Wide DFOV systems have the ability of portraying visual imagery and providing motion cues in the periphery, resulting in a greater sense of presence and self-movement, compared to more limited DFOV devices such as the desktop display [57]. However, it has been demonstrated that the wide DFOV systems are associated with greater levels of simulator sickness [58]. One possible reason is that distortions of visual and motion cues are magnified outside the center of projections on the HMD [59]. In contrast, the DFOV of the standard desktop display used in this study was small enough that dVFT users did not perceive motion in their visual periphery. This peripheral stability was consistent with the stability of the user’s body, and so might have reduced sensory conflict, thus reducing simulator sickness.

It is noteworthy that participants in the two VFT groups did not report any adverse effects associated with their participation beyond mild symptoms of simulator sickness. We have been experimentally exploring the possible side effects of an iVR experience; and our previous study has provided evidence that iVR stimulations may be overstimulating due to users’ prolonged exposure to virtual environments [34]. For this reason, we kept the virtual version of the field trip experience short (20 minutes) while making efforts to deliver teaching content comparable to the AFT through tailoring audio clips to the different field trip scenarios. Consequently, both dVFT and iVFT experiences were associated with minimal simulator sickness and were viewed as being realistic, smooth, valuable, and easy to use.

With regard to learning experience, although there is evidence for field trip enjoyment favoring the iVFT group, meaning that iVFT students might possess higher motivation than dVFT students to further explore the field site, our results indicate that learning geology in iVR was not more effective than the desktop display. In particular, students’ learning outcomes, quantified through the perceived learning effectiveness, field trip satisfaction, and in-VFT test scores, were not statistically different between the iVR and desktop versions of the VFT. Furthermore, the results show no significant differences in VR features and usability rated by participants between VFTs experienced in iVR and on the desktop display. These findings are not consistent with the study of Makransky et al. [19] indicating a negative effect of iVR on learning about lab procedures and lab equipment, but instead depict a similar image to what has been observed in previous studies of other learning topics. An example of this is the study by Moreno and Mayer [60] in which students learned a lesson of botany presented either on an HMD or via a desktop display. No differences in knowledge retention, transfer, and program ratings were found across media. Similarly, no differences emerged in knowledge acquisition and self-efficacy when testing an aviation safety training game played through an iVR system and a conventional desktop computer [20].

Nonetheless, results of the current study seem to be discordant with other previous literature, according to which better learning experience and performances were obtained when using iVR setups [28, 32, 36]. Such non-homogeneous results may be due to a variety of reasons. One possible explanation is that, given its heavy dependence upon 360° imagery, the current form of iVFT did not take the best advantage of iVR features. For instance, walking, as an important affordance of room-scale iVR applications, was not fully supported in our 360°-based iVFT experience. In addition, the Salona Formation is a linear outcrop along a straight highway, meaning that it does not actually require a 360° view for users to look around. There are other locations where a panoramic view would be beneficial such as a forest or the top of a mountain. Our results could also be explained by the possible different image qualities of dVFT and iVFT experiences. Participants in our past studies often mentioned the desire for higher image qualities. Leveraging the high-performance graphics cards, we were able to build 360° images with the highest resolution and render scene content using the fantastic level under Unity3D quality settings. Although dVFT and iVFT used the same quality settings, interestingly, participants who had experienced both types of VFTs in our pilot study indicated that 360° images rendered on the desktop display appeared to have higher resolution than on the Vive Pro HMD. Future study is necessary to clarify the effect of display type on the viewing experience for 360° imagery.

Despite the null effect for VR features, usability, and learning outcomes across VFT groups, iVFT participants demonstrated more positive attitudes toward VFTs after experiencing the Salona iVFT compared to their attitudes measured before the iVR intervention; such attitude change was not observed in the dVFT group. Using a desktop display as the medium to deliver learning content cannot be considered new at all; indeed, students can easily access numerous desktop-based field trip experiences through the internet using their own laptops (e.g., web-based VFTs developed by Arizona State University), so it is perhaps not surprising that dVFT participants did not change their VFT attitudes from pre- to post-measures. In contrast, iVR is a novel experience for most people, especially because it was not widely commercialized to the general public until 2016; this technology can elicit feelings of awe within laboratory conditions [12]. After the iVFT, the participants experiencing the novelty effect brought by the HTC Vive Pro might become aware of the potential of iVR

899

to facilitate learning about geology that they did not realize before, thus positively changing their attitudes toward VFTs, and possibly the field of geology.

The current study also employed a control group in which students went on the actual field trip (AFT) only. Results indicate that both dVFT and iVFT participants overall reported a better learning experience as well as more positive perceived learning outcomes than AFT participants. Specifically, the results show a positive response of students to the dVFT and iVFT with higher values for field trip enjoyment, cognitive benefits, perceived learning effectiveness, and field trip satisfaction, compared to students who went on the AFT. Our results essentially confirmed the findings of Klippel’s study [7], about higher ratings of field trip enjoyment and learning experience after an iVFT experience as compared to the actual ones.

Field trips in STEM-fields allow students to apply knowledge gained in lectures and laboratories and to inquire about phenomena and principles in-situ through exposure to a real setting. This includes the ability to discover through making observations, and sampling the real world, whether it be a plant in a biology course or a rock specimen in a geoscience course. The reality of fields trips; however, may not be perfect. Klippel et al. [7] identified a number of practical challenges of AFTs, that VFTs have the ability to overcome. We discuss these factors, as they pertain to our results of better learning experiences by VFT participants. These factors include: 1) distraction: students’ attention may have been directed by peers or busy traffic at Stop 2 along the highway; 2) number of instructors: there were two instructors per 25 students on each field trip, which may have led to less effective verbal instructions [61]; 3) weather: it was cold and rainy during at least one of the lab sections; 4) location: the outcrop of the Salona Formation, although an outstanding location for accomplishing the goals of the lab, is not the Grand Canyon (i.e., it may not awe students to the same extent), and therefore may not lead to a high enjoyment factor; and 5) participants: this course, although an introductory geoscience course, was dominated by engineering majors who were required to take it. We do not have specific student feedback regarding the first, second, fourth and fifth factors. Students did comment on factor three as a reason for lower enjoyment during the AFT.

5 LIMITATIONS AND FUTURE DIRECTIONS There are some potential limitations in the study design. First, the number of participants in each VFT group is relatively small. For this reason, the current study did not adjust the statistical inference while considering the set of measures simultaneously (i.e., multiple comparisons problem [62]). To account for multiple comparisons, we have listed all of the individual p values in Tables 2 and 3, and have reported their confidence intervals in Table 2 and following the significant results in Section 3.2. One thing to keep in mind is the increased type I error rate of these p values [63], so the results of this study should be interpreted with caution. Increasing the number of participants followed by multiple testing corrections can increase statistical power and provide further clarity by reducing the type I error for null associations. We plan to continue our testing, assessment, and evaluation of these VFT groups in subsequent semesters.

Second, the VFTs presented in the current study replicated the actual physical reality of the field site. Although there were some value added features such as the elevated perspective using images from tall tripods, VFT participants were still largely confined to the same physical constraints experienced during the AFT. Klippel et al. [7] have proposed three levels of VFTs—basic VFT (providing a virtual replication of a traditional experience), plus VFT (providing perspectives and information that cannot be accessed in the normal confines of physical reality), and advanced

VFTs (requiring extensive interactivity and the generation of models and simulations that can be manipulated to expand the number and kinds of ways a user can engage or query the field site). Inspired by Klippel’s taxonomy of VFTs, one of our future goals is to create more advanced VFT experiences to afford users access to otherwise inaccessible physical realities, such as observations of changes over geologic time, analysis of geologic data for synthesizing multiple lines of evidence, or a simulation that shows competing explanations for the formation of geological structures. An advanced VFT should also take advantage of the full potential of iVR through 3D reconstruction of real scenes that allows users to, for example, walk around the virtual field site and inspect virtual objects from up close. Achieving these goals will demand continued creation of 3D authentic models, development of rich interactivity within the VFT, and also creativity and care in learning design to scaffold higher-order thinking skills described in Bloom's taxonomy of educational objectives (i.e., analysis, evaluation, and creation [9, 64]).

Finally, students’ learning performance was not assessed in the comparison between VFT and AFT. The current study did not include the scores that students received in the official lab assignment for data analysis. This is because 1) more than half of the students did not provide consent to the release of their lab grades; 2) students were given one extra week to complete the lab assignment, meaning that they could have sought answers from other sources in addition to their field trip experiences; and 3) the lab assignment comprised description, discussion, and drawing questions; although there was a rubric, scores might vary due to the fact that they were assigned by different TAs. For these reasons, we are in the process of tailoring a set of multiple-choice questions testing skills and knowledge that will be suitable for both the VFT and AFT experiences in order to objectively and comprehensively measure students’ learning outcomes toward the field site.

6 CONCLUSION The significant positive effect of both dVFT and iVFT on students’ learning experience and perceived learning outcomes has provided empirical evidence of the potential of VFTs to support and enhance field-based learning in geoscience education. This finding is consistent with studies by Klippel et al. [7] and Mead et al. [9], indicating the effectiveness of VFTs in geoscience learning. Regarding immersion (desktop versus iVR), the current results provide evidence that the desktop is similarly efficient for supporting VFT experiences. No significant differences were found in VR features, system usability, learning experience, and performance of participants between the two immersion levels. iVR that supports fully immersive and embodied experiences may play an irreplaceable role in some learning situations (e.g., 3D data visualization and analysis: [65, 66]; procedural memorization tasks: [67, 68]), but at least for the range of learning objectives and activities in the current VFT, a standard desktop display with the mouse and keyboard seems to be sufficient to fulfil the needs for learning geologic principles and concepts, especially given that the iVR system is more expensive, less comfortable, and more difficult to set up.

ACKNOWLEDGMENTS We wish to thank Dr. Mark Patzkowsky for discussions regarding the geology of the Salona Formation, and Mr. Machel Higgins and Mr. Victor Garcia for their logistical help in performing the study. This study was funded through a Penn State Strategic Planning award. We Are! Dr. Klippel would also like to acknowledge funding for this work through the National Science Foundation grants #1617396 and #1526520.

900

REFERENCES [1] R. R. Compton, Geology in the Field. New York, Chichester,

Brisbane: John Wiley & Sons, 1985. [2] G. Dolphin, A. Dutchak, B. Karchewski, and J. Cooper, “Virtual

field experiences in introductory geology: Addressing a capacity problem, but finding a pedagogical one,” Journal of Geoscience Education, vol. 67, no. 2, pp. 114–130, 2019.

[3] G. Tuthill and E. B. Klemm, “Virtual field trips: Alternatives to actual field trips,” International Journal of Instructional Media, vol. 29, no. 4, pp. 453–468, 2002.

[4] R. Frodeman, Geo-logic: Breaking ground between philosophy and the earth sciences / Robert Frodeman. Albany, N.Y.: State University of New York Press, 2003.

[5] A. J. Watson, “The effectiveness of the GEOL336 Iceland virtual fieldtrip to aid student sketching and interpretation of lava flows,” Master's dissertation, University of Canterbury, 2019.

[6] F. D. Granshaw and D. Duggan-Haas, “Virtual fieldwork in geoscience teacher education: Issues, techniques, and models,” in Google Earth and Virtual Visualizations in Geoscience Education and Research: Geological Society of America Special Paper 492, S. J. Whitmeyer, J. E. Bailey, D. G. de Paor, and T. Ornduff, Eds., 2012, pp. 285–303.

[7] A. Klippel, J. Zhao, K. L. Jackson, P. La Femina, C. Stubbs, R. Wetzel, J. Blair, J. O. Wallgrün, and D. Oprean, “Transforming Earth Science Education Through Immersive Experiences: Delivering on a Long Held Promise,” Journal of Educational Computing Research, vol. 10, no. 2, 2019.

[8] J. Stainfield, P. Fisher, B. Ford, and M. Solem, “International Virtual Field Trips: A new direction?” Journal of Geography in Higher Education, vol. 24, no. 2, pp. 255–262, 2000.

[9] C. Mead, S. Buxner, G. Bruce, W. Taylor, S. Semken, and A. D. Anbar, “Immersive, interactive virtual field trips promote science learning,” Journal of Geoscience Education, vol. 67, no. 2, pp. 131–142, 2019.

[10] S. D. Hurst, R. D. Perkins, and J. A. Karson, “Interactive computerized geologic field trips as case studies for geologic instruction,” Geological Society of America, vol. 24, p. A131 (Abstracts with programs), 1992.

[11] S. D. Hurst, “Use of “virtual” field trips in teaching introductory geology,” Computers & Geosciences, vol. 24, no. 7, pp. 653–658, 1998.

[12] F. Pallavicini, A. Pepe, and M. E. Minissi, “Gaming in Virtual Reality: What Changes in Terms of Usability, Emotional Response and Sense of Presence Compared to Non-Immersive Video Games?” Simulation & Gaming, vol. 50, no. 2, pp. 136–159, 2019.

[13] M. Slater and S. Wilbur, “A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments,” Presence: Teleoperators and Virtual Environments, vol. 6, no. 6, pp. 603–616, 1997.

[14] H. L. Miller and N. L. Bugnariu, “Level of Immersion in Virtual Environments Impacts the Ability to Assess and Teach Social Skills in Autism Spectrum Disorder,” (eng), Cyberpsychology, behavior and social networking, vol. 19, no. 4, pp. 246–256, 2016.

[15] D. M. Markowitz, R. Laha, B. P. Perone, R. D. Pea, and J. N. Bailenson, “Immersive Virtual Reality Field Trips Facilitate Learning About Climate Change,” (eng), Frontiers in psychology, vol. 9, pp. 1–20, 2018.

[16] J. Zhao and A. Klippel, “Scale - Unexplored Opportunities for Immersive Technologies in Place-based Learning,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 2019, pp. 155–162.

[17] K. Kim, M. Z. Rosenthal, D. Zielinski, and R. Brady, “Comparison of desktop, head mounted display, and six wall fully immersive systems using a stressful task,” in 2012 IEEE Virtual Reality Workshops (VRW), Costa Mesa, CA, USA, 2012, pp. 143–144.

[18] K. Kim, M. Z. Rosenthal, D. J. Zielinski, and R. Brady, “Effects of virtual environment platforms on emotional responses,” (eng), Computer methods and programs in biomedicine, vol. 113, no. 3, pp. 882–893, 2014.

[19] G. Makransky, T. S. Terkildsen, and R. E. Mayer, “Adding immersive virtual reality to a science lab simulation causes more presence but less learning,” Learning and Instruction, 2017.

[20] F. Buttussi and L. Chittaro, “Effects of Different Types of Virtual Reality Display on Presence and Learning in a Safety Training Scenario,” (eng), IEEE transactions on visualization and computer graphics, vol. 24, no. 2, pp. 1063–1076, 2018.

[21] C. Dede, “Immersive interfaces for engagement and learning,” (eng), Science (New York, N.Y.), vol. 323, no. 5910, pp. 66–69, 2009.

[22] M. Matamala-Gomez, T. Donegan, S. Bottiroli, G. Sandrini, M. V. Sanchez-Vives, and C. Tassorelli, “Immersive Virtual Reality and Virtual Embodiment for Pain Relief,” (eng), Frontiers in human neuroscience, vol. 13, p. 279, 2019.

[23] K. Kilteni, R. Groten, and M. Slater, “The Sense of Embodiment in Virtual Reality,” Presence: Teleoperators and Virtual Environments, vol. 21, no. 4, pp. 373–387, 2012.

[24] S. J. G. Ahn, J. Bostick, E. Ogle, K. L. Nowak, K. T. McGillicuddy, and J. N. Bailenson, “Experiencing Nature: Embodying Animals in Immersive Virtual Environments Increases Inclusion of Nature in Self and Involvement with Nature,” J Comput-Mediat Comm, vol. 21, no. 6, pp. 399–419, 2016.

[25] J. Mestre, “Transfer of Learning: Issues and Research Agenda,” National Science Foundation, Washington, D.C., 2002.

[26] Y.-J. Lan, S.-Y. Fang, J. Legault, and P. Li, “Second language acquisition of Mandarin Chinese vocabulary: Context of learning effects,” Education Tech Research Dev, vol. 63, no. 5, pp. 671–690, 2015.

[27] W. Alhalabi, “Virtual reality systems enhance students’ achievements in engineering education,” Behaviour & Information Technology, vol. 35, no. 11, pp. 919–925, 2016.

[28] E. Krokos, C. Plaisant, and A. Varshney, “Virtual memory palaces: Immersion aids recall,” Virtual Reality, vol. 23, no. 1, pp. 1–15, 2019.

[29] J. Legault, J. Zhao, Y.-A. Chi, W. Chen, A. Klippel, and P. Li, “Immersive Virtual Reality as an Effective Tool for Second Language Vocabulary Learning,” Languages, vol. 4, no. 1, p. 13, 2019.

[30] J. Parong and R. E. Mayer, “Learning science in immersive virtual reality,” Journal of Educational Psychology, vol. 110, no. 6, pp. 785–797, 2018.

[31] J. Sweller, J. J. G. van Merrienboer, and F. G. W. C. Paas, “Cognitive Architecture and Instructional Design,” Educational Psychology Review, vol. 10, no. 3, pp. 251–296, 1998.

[32] R. A. Ruddle, S. J. Payne, and D. M. Jones, “Navigating Large-Scale Virtual Environments: What Differences Occur Between Helmet-Mounted and Desk-Top Displays?” Presence: Teleoperators and Virtual Environments, vol. 8, no. 2, pp. 157–168, 1999.

[33] H. Li and N. A. Giudice, “The effects of immersion and body-based rotation on learning multi-level indoor virtual environments,” in ISA 2013: Proceedings of the Fifth ACM SIGSPATIAL International Workshop on Indoor Spatial Awareness: November 5, 2013, Orlando, Florida, USA, 2013, pp. 8–15.

[34] Zhao, Jiayan, T. Sensibaugh, M. Minear, B. Bodenheimer, T. McNamara, A. Nazareth, N. Newcombe, and A. Klippel, “Desktop Versus Immersive Virtual Environments: Effects on Spatial Learning,” Spatial Cognition & Computation, in review.

[35] G. Mineev, “The Impact of Immersive Virtual Reality on Effectiveness of Educational Games,” Master's Thesis, Department of Information and Computing Sciences, Utrecht University, 2017.

[36] B. An, F. Matteo, M. Epstein, and D. E. Brown, “Comparing the Performance of an Immersive Virtual Reality and Traditional Desktop Cultural Game,” in 2nd International Conference on Computer-Human Interaction Research and Applications, Seville, Spain, 2018, pp. 54–61.

[37] J. I. Spicer and J. Stratford, “Student perceptions of a virtual field trip to replace a real field trip,” Journal of Computer Assisted Learning, vol. 17, no. 4, pp. 345–354, 2001.

[38] C. Deng, Z. Zhou, W. Li, and b. Hou, “A Panoramic Geology Field Trip System Using Image-Based Rendering,” in COMPSAC 2016 Proceedings, Atlanta, Georgia, USA, 2016, pp. 264–268.

901

[39] Unity®, Unity3D: Unity Technologies, 2019. [40] J. Zhao, M. Simpson, J. O. Wallgrün, P. Sajjadi, and A. Klippel,

“Exploring the Effects of Geographic Scale on Spatial Learning,” Cognitive Research: Principles & Implications, in revision.

[41] A. Klippel, J. Zhao, P. Sajjadi, J. O. Wallgrün, D. Oprean, and M. Bagher, “Immersive Place-based Learning – An Extended Research Framework,” in 2020 IEEE Virtual Reality Workshop on K-12 Embodied Learning Through Virtual and Augmented Reality (KELVAR), In review.

[42] A. Klippel, J. Zhao, D. Oprean, J. O. Wallgrün, C. Stubbs, P. La Femina, and K. L. Jackson, “The value of being there: Toward a science of immersive virtual field trips,” Virtual Reality, vol. 1, no. 4, p. 24, 2019.

[43] A. Klippel, D. Oprean, J. Zhao, J. O. Wallgrün, P. LaFemina, K. L. Jackson, and E. & Gowen, “Immersive learning in the wild – a progress report,” in 5th Annual Immersive Learning Research Network Conference, 2019.

[44] J. O. Wallgrün, J. S.-K. Chang, J. Zhao, P. Sajjadi, D. Oprean, T. B. Murphy, J. Baka, and A. Klippel, “For the Many, Not the One: Designing Low-Cost Joint VR Experiences for Place-Based Learning,” in Virtual reality and augmented reality: 16th EuroVR International Conference, EuroVR 2019, Tallinn, Estonia, October 23–25, 2019: proceedings / Patrick Bourdot, Victoria Interrante, Luciana Nedel, Nadia Magnenat-Thalmann, Gabriel Zachmann (eds.), 2019, pp. 123–148.

[45] AgiSoft, PhotoScan Professional (Version 1.4.3) (Software), 2018. [46] M. Hegarty, “Development of a self-report measure of

environmental spatial ability,” Intelligence, vol. 30, no. 5, pp. 425–447, 2002.

[47] H. Jeffreys, Theory of probability, 3rd ed. Oxford Oxfordshire: Clarendon Press; New York: Oxford University Press, 1961.

[48] R. S. Kennedy, N. E. Lane, K. S. Berbaum, and M. G. Lilienthal, “Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness,” The International Journal of Aviation Psychology, vol. 3, no. 3, pp. 203–220, 1993.

[49] A.-L. E. Lee, K. W. Wong, and C. C. Fung, “How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach,” Computers & Education, vol. 55, no. 4, pp. 1424–1442, 2010.

[50] P. Vorderer et al., MEC Spatial Presence Questionnaire (MECSPQ): Short Documentation and Instructions for Application. [Online] Available: https://academic.csuohio.edu/kneuendorf/frames/MECFull.pdf. Accessed on: Sep. 27 2017.

[51] W. Wirth, T. Hartmann, and S. Böcking et al., “A Process Model of the Formation of Spatial Presence Experiences,” Media Psychology, vol. 9, no. 3, pp. 493–525, 2007.

[52] R. Lenth, Emmeans package: Estimated Marginal Means, aka Least-Squares Means. R package version 1.3.3. Available: https://CRAN.R-project.org/package=emmeans. Accessed on: Jun. 30 2019.

[53] M. Dawson, “Chapter 6: Embedded and situated cognition,” in Routledge handbooks, The Routledge handbook of embodied cognition, L. Shapiro, Ed., 1st ed., London, New York: Routledge, Taylor & Francis Group, 2014, pp. 59–67.

[54] Y.-J. Lan, N.-S. Chen, P. Li, and S. Grant, “Embodied cognition and language learning in virtual environments,” Education Tech Research Dev, vol. 63, no. 5, pp. 639–644, 2015.

[55] J. Moss, J. Scisco, and E. Muth, “Simulator Sickness during Head Mounted Display (HMD) of Real World Video Captured Scenes,” Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 52, no. 19, pp. 1631–1634, 2008.

[56] J. D. Moss and E. R. Muth, “Characteristics of head-mounted displays and their effects on simulator sickness,” (eng), Human factors, vol. 53, no. 3, pp. 308–319, 2011.

[57] P. J. Sparto, S. L. Whitney, L. F. Hodges, J. M. Furman, and M. S. Redfern, “Simulator sickness when performing gaze shifts within a wide field of view optic flow environment: preliminary evidence for using virtual reality in vestibular rehabilitation,” (eng), Journal of neuroengineering and rehabilitation, vol. 1, no. 1, p. 14, 2004.

[58] H.-M. Huang, U. Rauch, and S.-S. Liaw, “Investigating learners’ attitudes toward virtual reality learning environments: Based on a

constructivist approach,” Computers & Education, vol. 55, no. 3, pp. 1171–1182, 2010.

[59] R. Pausch and T. Crea, “A Literature Survey for Virtual Environments: Military Flight Simulator Visual Systems and Simulator Sickness,” Computer Science Report No. TR-92-25, http://www.cs.cmu.edu/~stage3/publications/93/journals/PRESENCE/militaryFlight/paper.html, 1992.

[60] R. Moreno and R. E. Mayer, “Learning science in virtual reality multimedia environments: Role of methods and media,” Journal of Educational Psychology, vol. 94, no. 3, pp. 598–610, 2002.

[61] R. G. Ehrenberg, D. J. Brewer, A. Gamoran, and J. D. Willms, “Class Size and Student Achievement,” (eng), Psychological science in the public interest: a journal of the American Psychological Society, vol. 2, no. 1, pp. 1–30, 2001.

[62] J. H. McDonald, Handbook of Biological Statistics, 2nd ed. Baltimore, Maryland, USA: Sparky House Publishing, 2009.

[63] S.-Y. Chen, Z. Feng, and X. Yi, “A general introduction to adjustment for multiple comparisons,” (eng), Journal of thoracic disease, vol. 9, no. 6, pp. 1725–1729, 2017.

[64] B. S. Bloom, Taxonomy of educational objectives. New York: Longmans Green, 1964.

[65] O. Kreylos, O. G. Staadt, D. Y. Sumner, G. Bawden, T. Bernardin, M. I. Billen, E. S. Cowgill, R. D. Gold, B. Hamann, M. Jadamec, and L. H. Kellogg, “Enabling scientific workflows in virtual reality,” in Proceedings VRCIA 2006: ACM International Conference on Virtual Reality Continuum and its Applications, Chinese University of Hong Kong, June 14-17, 2006, Hong Kong, China, 2006, pp. 155–162.

[66] M. Simpson, J. Zhao, and A. Klippel, “Take a walk: Evaluating Movement Types for Data Visualization in Immersive Virtual Reality,” in Proceedings of Workshop on Immersive Analytics (IA), 2017.

[67] A. Sowndararajan, R. Wang, and D. A. Bowman, “Quantifying the benefits of immersion for procedural training,” in Proceedings of the 2008 Workshop on Immersive Projection Technologies/Emerging Display Technologies, 2008, pp. 1–4.

[68] E. D. Ragan, “The Effects of Higher Levels of Immersion on Procedure Memorization Performance and Implications for Educational Virtual Environments,” Presence: Teleoperators and Virtual Environments, vol. 19, no. 6, pp. 527–543, 2010.

902