assessment of teaching and learning with technology at seton hall university janet easterling,...
TRANSCRIPT
Assessment of Teaching and Learning with Technology at
Seton Hall UniversityJanet Easterling, Institutional Research
Danielle Mirliss, Senior Instructional DesignerShayle Abelkop, Instructional Designer
Copyright Abelkop, S., Easterling, E., and Mirliss, D. (2007). This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.
Why Mobile Computing
“Seton Hall University’s Mobile Computing Program is designed to break down the
barriers of space and time and allow all our students to fully participate in our scholarly
community and achieve high levels of performance.”
Mobile Computing Mission Statement
Mobile Computing Timeline
• 1994 – 1996: Development of SHU Strategic Plan
• 1995 – 1997: Mobile Computing Pilot Project with small cohorts of students in Business School
• 1997: Mobile Computing Pilot Project expands to 300 incoming Freshman
• 1998: Mobile Computing Program launched for all incoming Freshman
• 2001: Mobile Computing Program expanded to select graduate programs; campus wireless network implemented
• 2002: SHU graduates first mobile class
• 2004: Tablet PC Pilot Project
Mobile Computing @ SHUInnovative academic program involving
three components:• Access: The University licenses student use
of a laptop computer as part of tuition and fees.
• Curricular Integration: The University encourages and supports faculty use of technology in teaching and learning.
• Network and Support Services: The University provides the infrastructure that enables effective use of technology in teaching and learning.
Mobile Computing Assessment• Began in 1996
• Student Survey
• Historically focused on:• Overall Satisfaction• Use of the Technology• Impact on Effective Teaching Practice
• 2005-2006 Renamed Teaching and Learning with Technology Survey• Emphasis on Student Use and Effective
Integration of Technology
Early Efforts/Challenges
• In an effort to assess everything the survey grew to monstrous proportions
• Felt a need to document clearly positive outcomes of efforts to infuse
• Then compelled to monitor positives
• Added questions but also kept all the questions…
Better but not there yet…
• Survey questions were narrowed to emphasize teaching and learning
• Exploratory approach to achieve better questions– open-ended/focus groups
• Smaller number, better questions
• Application of results to inform practice
Where we want to be…• Ask questions that are meaningful to
all stakeholders (even students!!)
• Craft a survey each year based on needs in a changing context
• Disseminate information to campus and beyond
• Include additional stakeholders to drive questions
Results from along the way…
Quantitative – student responses on usage, gains, preferences & benefits
Qualitative – opinions, details of perceived benefits and barriers
T & Learning with Technology Students
1. Hours each week on-line Q3
2. Usage of IT to .. Q4
3. IT preferences in courses Q5
4. IT improves collaboration, feedback, control? Q6
5. IT improves my learning Q7
6. IT feature most valuable Q8
7. Gains d.t. on-line Bb/web Q12
8. Opps 7prins qual learning Q13
Quantitative highlights
Learning with TechnologyStudent Survey
• administered April 2006 to a random sample of freshmen and juniors
• oversampling for males and for students identifying as non-white
• 253 responses, overall response rate 30%
• Higher rates for Women, Residents, FR• Lower rates for Men, Non-White,
Commuting Students, JR
1 –. Hours each week on-line Q3How many hours per week would you estimate you spend…
Most frequent response 6-10 hours / week on-line (www/e-mail/..) for
academic work avg 11 hours
~1 in 2 students (a little under) specify more than 10 hours a week ~1 in 5 students (a little over) 5 or fewer hours a week ~1 in 10 students (a little over) 21 or more hours a week
• 45% more than 10 hours (11% 21+)• 23% 5 hours or less ( 4% zero)
1 –. Hours each week on-line Q3How many hours per week would you estimate you spend ….
Most frequent response 6-10 hours / week on-line (www/e-mail/..) for
non-academic work avg 13 hours
~1 in 2 students (a little over) specify more than 10 hours a week ~1 in 4 students 5 or fewer hours a week ~1 in 4 students (a little over) 21 or more hours a week
• 55% more than 10 hours (27% 21+ hours)• 25% 5 hours or less (11% zero)
1 –. Hours each week on-line Q3
How many hours per week would you estimate you spend..
Summed responses for hours / week on-line (www/e-mail/..) in total avg 23 hours
~3 in 5 students specify 21 or more !
~1 in 4 students 10 or fewer
~1 in 10 students 41 or more !!
• 60% more than 20 hours (12% 41+ hours)• 25% 10 hours or less ( 2% 5 or less)
2 – Use of IT for Academic work Q4 On average how often do you use Information Technology to.. Daily/Weekly/Monthly/Rarely/Never
Frequency of use of IT for academic work
• to complete course assignments 77% D/W
• e-resources from Library web 48% D/W• e-resources from other sites 72% D/W
(other than a library web site)
2 – Use of IT for Academic work Q4 On average how often do you use Information Technology to.. Daily/Weekly/Monthly/Rarely/Never
Freq of use of IT for academic work
to work in teams
• during class 62% D/W/M
• outside of class 70% D/W/M
(to complete course assignments)
3 – Student prefs for use of tech in courses Q5 I prefer taking courses that …
first choice, courses that• use a moderate level of technology
42% (email, several ppt presentations, some on-line activities/content)
however, almost tied ! • use technology extensively
39% (class lecture notes on-line, ppt, computer simulations, streaming video)
third choice• use limited technology
13% (professor email, limited ppt use)
4 – Impact of IT Q6 Compared to courses that use little or not technology, the use of technology … (agreement)
use of IT allows me toTake greater control of my course activities
planning, apportioning time, awareness of success/failure 89% (42% strongly agree, 37% agree)
use of IT results inMore prompt feedback from my instructor
85% (40% strongly agree, 45% agree)
use of IT helps me Better communicate and collaborate with my classmates
80% (38% strongly agree, 42% agree)
5 –IT Improves Learning? Q7 The use of technology in my courses has improved my learning… (agreement)
use of IT in my Courseshas Improved my Learning
80% (strongly agree or agree)------------------------------------------------------------------
21% strongly agree 59% agree
15% neutral
3% disagree 2% strongly disagree
6 – IT Most valuable benefit Q8 Which of the following benefits from using technology in your classes are most valuable to you…
single most valuablebenefit from using technology
68% convenience / saves time
19% helps manage class activities
9% improves my learning
3% other 2% none
7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)?
Course selected by the student as one that makes effective use ..
(student reasons for selecting – Danielle Mirliss will present these)
7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)?
In course selected by the student as one that makes effective use ..
Improved Learning %responses Online readings & links to materials 67% Online discussion board 47% Sharing materials with students 45%
Improved Management of Class Activities Online syllabus 85% Ability to turn in assignments on-line 73%
7 – Perceived gains from on-line (Bb/web) use Q12 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, what was the effect of having the following on-line features (available through Blackboard or your professor’s web site)?
In course selected by the student as one that makes effective use ..
Positive Experience 0/1 Mean Online syllabus .89 Online readings & links .86 Online turn in assignments .79 Online sharing materials .70 Online get grades, comms .66 Online track your grades .64 Online discussion board .63
8 – Opportunity for 7-principles learning env’s Q13 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, how often were you able to do or experience the following)?
In course selected by the student how often able to do or experience ..
Student Faculty Contact %often {very} Have contact with professors 80% on course content
Diverse ways of learning % often {very}
Explore course materials .. 75% Demonstrate what I have learned .. 63% .. in different ways
8 – Opportunity for 7-principles learning env’s Q13 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, how often were you able to do or experience the following)?
In course selected by the student how often able to do or experience ..
Cooperation among students %often {very}
Spend time with other students on .. 55%
Feedback Get prompt feedback on performance 73% Observe & record my own progress 57%
Time on Task Edit and improve my projects / assign. 60%
8 – Opportunity for 7-principles learning env’s Q13 As you answer the following questions, think about one particular course from among those you are taking this semester that, in your opinion, makes effective use of technology to support teaching and learning… In this course, how often were you able to do or experience the following)?
Does positive experience with related on-line activity relate to frequency of opportunity for high quality learning environments?
Cooperation among students %often {very} Q13b
Spend time with other students on .. 55%
Positive experience with on-line Q12Online readings & links .86 .23 Online sharing materials .70 .32 Online discussion board .63 .14 mean correlation with 13b
9 – Strengths, Challenges Quantitative data
More gen’l observations .. come back to..
Strengths .. results s-f contact .. (as in prior years) s-s interaction .. (newly revealed)
Challenges .. data ? serious responses .. (given incentive) ? generalizable .. (given response %)
10 –Teaching & Learning with Technology survey assessment continued
Next ..
Reviews of qualitative - Danielle & Shayle helps .. hinders .. effective use ..
+Focus Groups to probe, explore - “effective uses” of technology – examps .. - gains from interactions – which online resources..
Survey Qs next cycle that probe effective use in terms meaningful to student
Qualitative Responses
Technology helps learning by…
• Providing anytime, anywhere access • Organizing information• Facilitating communication among students/professors• Broadening research/resource materials• Efficient methods of submitting assignments• Making access equitable• Having projectors in class• Exercising technical skills
What students said…• “Giving us access to certain resources that benefit our learning experience as a whole”• “Giving students an outlet to learn valuable skills”• “Making it easier to contact professors and fellow students for help, notes, etc.” • “Technology is always available and accessible when I need it, whether I want to take notes in class with my laptop or use resources available through the school’s homepage”• “With all the information that is now provided via the internet, the technology we have guides us to easy access for attaining this information.”
Technology prevents learning by…
• Providing distractions
• Technical issues• Too much reliance • Facilitating plagiarism• Less personal contact w/ students
“At Seton Hall, technology can prevent learning because during lectures some students may be
too busy using instant messenger instead of paying attention to the professor.”
What students said…• “Being a distraction for those who do not know when to separate from it at needed times.”• “Having access to cheating methods.” • “Technology can prevent students from learning because they are able to surf the net while in class as well as talk on AIM.” • “Many students bring their laptops to classes and become distracted by them because it gives them something to do besides paying attention.”
Why did you choose this course as one that makes effective use of
technology?• Assignments – ability to complete and submit assignments electronically
“This course not only provides me with a syllabus online but with other assignments that are to be done online.”
• Collaboration – use of technology to communicate and co-author assignments
“I thought it allowed me to work better with my classmates.”
• Convenience – enables students to keep track of work and updated materials
“Kept me up to date with my course grades and assignments.”
• Feedback – provides a way for faculty to give feedback on assignments
“Many of our assignments are turned in online via our discussion board and we are given feedback on certain assignments from our professor as well.”
Why did you choose this course as one that makes effective use of
technology?
• Frequency – course uses technology more than other courses “Because technology is used for every
aspect of the class: assignments, quizzes, lectures, discussion board, etc.”
• Information Literacy – ability to access resources from library and Internet “We use the library database every day.”
Why did you choose this course as one that makes effective use of
technology?
• Learning – contributed to students’ understanding of course material “Overall the technology use enhanced my
level of learning in the course.” • Pedagogy – effective faculty use of
technology “The professor uses power point
presentations to teach us the material, and it makes it easier to follow along and catch up on missed topics.”
Why did you choose this course as one that makes effective use of
technology?
• Technology Interest – fulfilled students’ interest in using technology “This course played into both my love of
technology and my major”
• Blackboard – frequently mentioned as tool employed in technology enhanced courses “All quizzes and notes are provided on
BlackBoard”
Why did you choose this course as one that makes effective use of
technology?
