a small-scale evaluation of adobe connect as a university web-conferencing tool

7/29/2019 A Small-Scale Evaluation of Adobe Connect as a University Web-Conferencing Tool

1/37

A Small-Scale Evaluation of

Adobe Connect as a UniversityWeb-Conferencing Tool

Jeff ToorongianEDTECH505-4172

August 1, 2011


2/37

2

Table of Contents

Learning Reflection. 3

Executive Summary. 4

Purpose of Evaluation.. 5Central Questions. 5Stakeholders. 6

Background Information.. 7Origins of Web Conferencing.. 7Adobe Connect Product Characteristics... 7People Involved with Adobe Connect.. 8Program Goals.. 8

Evaluation Design.... 9Evaluation Development Process. 9Survey Design.. 10Usage Characteristics... 10Teaching and Learning Tools... 11User Feedback.. 11Interview... 12

Results and Discussion. 12Survey Data Reliability 12Usage Data... 13

Adobe Connect Features.. 15Adobe Connect Satisfaction Level and Areas for Improvement. 16Interview Responses. 17

Conclusions and Discussion. 18Immediate Conclusions 18Long-Range Planning.. 19Evaluation Insights.. 20

Appendix A: Adobe Connect Response Survey.. 21Appendix B: Adobe Connect Survey Questions.. 29

Appendix C: Interview Responses (Adobe Connect Administrator)... 33Appendix D: Adobe Connect Server Usage Report (2010-11) 35Appendix E: Evaluators Program Description 36Appendix F: Evaluation Project Timeline 37


3/37

3

Learning Reflection

As I have progressed through this course the systematic nature of evaluation hasbecome clearer to me. The evaluator typically begins by determining the reason or needfor the evaluation. Is it a requirement of a program? Is there a need to establish

effectiveness to validate a programs existence? Does a program need a tune-up, or away to improve efficiency? Does the program or product have lasting impact on thosewho used or participated in it? Efficiency, effectiveness and impact are the three levelsof evaluation that are validated. Those responsible for the program, the stakeholders,should be at the table during the evaluation process. The evaluator works withstakeholders to understand the program, the identified needs that it aims to serve, its goalsand objectives, and the process established to achieve the desired outcomes. Thisinformation serves as the basis for an evaluators program description (EPD). The EPDcan assist the evaluator in the development of an evaluation design. The evaluationdesign also identifies the information needed to determine whether goals are beingmet. It describes the appropriate data sources and the related data gathering instruments

that will be used to gather the information. The evaluation design is the blueprint which,when implemented, results in a final report that objectively reflects on the relativesuccess of a program, identifies areas where problems may exist, and makesrecommendations on how to improve things.

In this final evaluation project I had a chance to engage this systematic processduring a review of a real project at my institution. One aspect that became very clear tome was that thorough and effective evaluations take time. It was not feasible to conducta full-scale evaluation due to the short duration of this accelerated summer course.Regardless of the time constraints, I was able to get my feet wet in the evaluationprocess. As an educational technologist the skills gleaned in this course will serve me

well in my professional work. There are many campus teaching and technologyinitiatives, usually with lofty companion goals and objectives, aimed at better servingfaculty and students in their teaching and learning endeavors. Often the core of theinitiative is an expensive implementation of technology tools. Frequently missing fromthis process is an analysis of the real needs that are present, an explanation of clearlydefined goals to be met, and a process of formative and summative evaluation necessaryto identify areas needing change during the process or upon its conclusion. Also missingis an established, impartial yardstick, to be used by program personnel and evaluators todetermine how the programs efforts are measuring up to defined criteria. Referencecriteria are critical in establishing credibility to evaluation results andrecommendations. Knowing these criteria in advance can help program stakeholders tune

their processes and procedures for maximum long-term impact. With my improvedunderstanding of the evaluation process I feel better equipped to help serve my institutionwith objective and systematic analyses of future campus educational initiatives. In theprocess I will be able to help demonstrate what products have been effective, whatprocesses may lack efficiency, and what programs have proven impact.


4/37

4

Executive Summary

As online and blended learning efforts increase at Michigan TechnologicalUniversity, the availability of a robust, synchronous web conferencing tool is critical to

the success of these university eLearning activities. This evaluation report details effortsto characterize the adoption rate, usage characteristics, areas of weakness, and overallimpressions of Adobe Connect web conferencing software by instructional faculty andresearchers. Scheduling limitations necessarily limited the scope and magnitude of thissmall-scale effort but its findings could serve as the foundation of any future campuswide evaluation of web conferencing activity. A targeted sample of Adobe Connect userswere invited to participate in an online survey which asked a series of questions related tohow often they used Adobe Connect, where they used it, what types of activities theyperformed with it, and the size of their virtual meeting groups. An interview with theAdobe Connect campus administrator was also conducted to obtain information on thehistorical use of web conferencing on campus, details on the selection of Adobe Connect

to replace the previous web conferencing tool, support and training activities on campus,and server-provided usage statistics.The survey response rate was sixty-seven percent, with all respondents indicating

either weekly or monthly use of Adobe Connect. Collaboration was indicated as the mostcommon web conferencing activity, both with students in a teaching role, and withprofessional colleagues in a research role. Recent usage reports indicated sixty-sevennamed organizers, approximately 325 active users, and more than 12,000 meeting hoursof activity at the university in the last academic year. Most surveyed users identifiedsmall group meetings (2-9 people) as their most common activity. All participantsindicated the use of audio in their meetings, while only fifty percent used video in all oftheir meetings. Almost all respondents used the share pod and chat pod, but the poll podand question pod were rarely used. The most common technical problem involvedaudio. Specific complaints included audio delay, audio distortion, and problems turningtheir microphone on or off. All survey participants indicated that local technical supportand training resources were somewhat or very important. The Adobe Connect systemadministrator echoed this importance of local training. Evaluation recommendationsinclude continued and enhanced local training and support, an investigation into optionsto host the Adobe Connect server on campus, and the need for central IT to determinehow Adobe Connect can integrate into existing enterprise applications and the newlearning management system that will be identified for campus in the next 12-18 months.


5/37

5

Purpose of Evaluation

This evaluation is intended as a preliminary effort to identify current usagecharacteristics, satisfaction levels, and overall impressions of Adobe Connect, thesupported web-conferencing software for teaching and learning activities at MichiganTechnological University. A full-featured, stable, and intuitive web-conferencingapplication is increasingly important in the daily activities of faculty, researchers, andstudents. Supporting the practice of web-conferencing and the specific application thatprovides this functionality has become a priority. Information obtained in this evaluationcan serve as the bridge to a larger, campus-wide analysis of web conferencing in thefuture. Adobe Connect has been used at Michigan Tech since the fall of 2008, and iscurrently offered as an Adobe-hosted solution.Central Questions

The primary questions guiding this evaluation process were intended to bettercharacterize the usage patterns, activities conducted, satisfaction levels, and identifiedareas of weakness of Adobe Connect, as indicated by university instructors andresearchers.Usage Characteristics

It is important to identify how users engage with Adobe Connect. To what extentare they utilizing its features and capabilities? How often do they use web-conferencing? Adobe Connect offers a full suite of web-conferencing tools includingaudio/video sharing, text chatting, questions and polling, and slide, whiteboard, desktopand application sharing. These features allow users to establish custom collaboration

environments to meet their needs. As more features are used and participant numbersincrease, the opportunities for problems also increases. A better understanding of userpreferences will allow for the development of a more comprehensive support structure forweb-conferencing on campus.

Web-Conferencing ActivitiesDetermining the most common types of web-conferencing activities is equally

important. These activities can include meetings, software demonstrations and training,office hours, exam or homework review sessions, presentations, and researchcollaborations. Each of these activities utilizes specific capabilities of the application, and

can require additional peripherals connected to users computers, and infrastructureenhancements in the various locations of the event. For example, a simple two-persongroup meeting could occur in respective office locations and require no more than acomputer with a web cam, and a headset with a microphone. More involved researchcollaboration might require desktop sharing, slide sharing, and chatting, in addition toaudio and video sharing. It might also involve more participant locations in differenttime zones and with different languages in use. These complications can greatly increasethe chances for problems. Understanding the types of activities and the associated needs


6/37

6

of the Adobe Connect users will help support groups to develop best practices, andcustomize training needs for these users.User Satisfaction

User satisfaction is another criteria we sought to better characterize through thisevaluation. Were there recurring frustrations identified by web-conferencing participantsduring their use of Adobe Connect? What types of activities were difficult, problematic,or not possible? Identifying a user-defined satisfaction level, even a broadly definedone, can help establish basic assumptions about the success of Adobe Connect insupporting user needs and helping to identify areas for improvement.

Areas of WeaknessRelated to satisfaction levels is the need to define areas where Adobe Connect is

perceived as ineffective or problematic amongst users. Are there specific functions that

are consistently identified as difficult to use or simply not functional? For example, dousers consistently report problems with voice over Internet audio (VoIP)? It is not theintent of this evaluation to determine the cause of technical problems, either user-centered or product-centered. Instead, identifying specific areas of weakness asidentified by the user-base will help stakeholders make improvements in theimplementation and support processes for Adobe Connect.

StakeholdersPrimary stakeholders for this evaluation are university personnel who are

responsible for supporting teaching and learning efforts on campus. Among these are theeLearning staff who engage directly with instructors for both pedagogical and technicalsupport of their collaborative efforts, and central IT members responsible for the network,classroom infrastructure, and administrative tasks associated with AdobeConnect. Evaluation results may help clarify usage patterns, identify areas whereimproved integration in a particular learning space is indicated, and help support unitplans for improved and expanded training opportunities for campus users. Additionalstakeholders are current users of Adobe Connect, and those who are considering itspotential value in their teaching and learning efforts. For current users, the evaluationresults may spawn new approaches to using Adobe Connect or may encourage thecommunity of users to better engage each other regarding best practices and lessonslearned with Adobe Connect.


7/37

7

Background InformationMichigan Technological University, located in the scenic western upper peninsula

of Michigan, has approximately 7,000 campus-based students. The majority of studentsstudy engineering and science disciplines, with programs in forestry, business, and

technology also prominent. The school currently has over 1,200 graduate students.

Origins of Web-Conferencing and Previous Products at Michigan TechThe first centrally supported web-conferencing application at Michigan Tech was

a product called Marratech. Like many of its competitors, Marratech featured a standardsuite of web-conferencing tools and functionality. The product originated in Europe, andwas discovered by a faculty member during external collaborative efforts with industrysponsors. It was particularly adept at quality voice over Internet (VoIP) support. TheeLearning services group installed the Marratech server in 2005. Before this installationthere were scattered pockets of web-conferencing activity in various academic and

research units, but it was done at the local level, without centralized support for theserver, training, administrative tasks, or user support. Marratech used a client-basedmodel, which required all users to download and install the client application on anycomputer they wished to use. Client support for Windows and Macintosh computers wasgenerally successful, but support for Linux users was always difficult. Though Linuxusers were a minority on campus, finding a solution that supported all platformseffectively was a goal. Google acquired Marratech in 2007. When it was apparent thatno further development of the product was planned the decision was made to identify anew web conferencing solution. Michigan Tech began testing Adobe Connect (knownthen as Adobe Acrobat Connect Pro) in the fall of 2008. In the spring of 2009, theMarratech server was turned off and all users were directed to Adobe Connect for their

web conferencing needs.

Adobe Connect Product CharacteristicsMichigan Tech currently uses the Adobe-hosted Connect web-conferencing

application. At the time of implementation a hosted solution was determined to be thequickest and most cost effective way to transition to a new web-conferencingproduct. From a user perspective, there is little difference between an Adobe-hosted

meeting and a meeting from a local server. Unlike Marratech, which required a clientapplication, Adobe Connect uses Flash to deliver meetings within a standard webbrowser. Simply clicking on a provided meeting URL allows a user to join a meeting.Because Flash is the delivery technology behind Adobe Connect, any computer operatingsystem that can run Flash can also run an Adobe Connect meeting without the need for alocal client. This allows Windows, Macintosh and Linux users to all participate in AdobeConnect meetings client-free.


8/37

8

Adobe Connect is a full-featured web conferencing application providing basicfunctions like support for both VoIP (voice over Internet) audio and phone-based audio,video, enhanced chat and whiteboard, polls, and question and answerfunctionality. Participants can also share their slides, desktop display, or specificapplications, and they can upload other audio or video content for use in a meeting.

Adobe Connect allows users to design custom meeting layouts, and integrate interactivequizzes or simulations to further enhance the collaborative experience.

People Involved with Adobe Connect

Adobe Connect is available to all members of the Michigan Techcommunity. Because the product is currently hosted on Adobe servers, users must firstrequest an account as a meeting organizer from the central IT organization. Once theyhave secured their account they are able to schedule meetings as often as they wish. Acurrent stipulation of the hosted solution limits any organizer to one concurrent meeting

event, with a maximum 100 participants. To date, these restrictions have not beenproblematic to most users. A recent report from the campus Adobe Connectadministrator indicated more than 300 current users and 144 individual meetinghosts. Users are defined as people who have entered an Adobe Connect meeting withtheir campus credentials. Actual user numbers are likely higher than this since peoplecan enter some Adobe Connect meetings without logging in, if the meeting host allowsthis. During the last academic year 382 unique Adobe Connect meeting rooms werecreated by campus users. Meeting rooms are often used for multiple instances of ameeting event. In this way, a meeting room could be used each week for a user groupinstead of having to create new meetings for each event.

Users include members of the faculty, staff, and student populations oncampus. Adobe Connect is used for eLearning activities like collaborative meetings,group projects, review sessions, and faculty office hours. Others use Adobe Connect forresearch collaboration or training presentations with colleagues around the world.Administrative functions like account creation, billing, and training reside in the IT andeLearning groups on campus. Actual Adobe Connect meetings occur in offices,conference rooms, classrooms, and private residences. At this time, meeting organizers(or their home departments) pay the annual license fee, and users are also responsible forobtaining the peripherals (web-cams and headsets) often necessary for meetingparticipation.

Program Goals

Prior to the move to Adobe Connect, the university was mindful of three primaryobjectives as it implemented its first campus-wide web-conferencing product. There wasa need for increased collaboration in classes. The campus also needed a synchronousonline tool with recording capability to connect learners and researchers both on and offcampus. Finally, there was a desire to select a synchronous tool that would be centrallysupported and could replace the numerous web-conferencing tools that were in scattereduse across campus on an ad hoc basis.


9/37

9

1) Increased Collaboration Capabilities

A recognized need to encourage more synchronous collaborative activitiesfor online, blended, and face-to-face classes was identified. Instructorsincreasingly demanded ways to engage their students outside the confines of theirclassrooms. For face-to-face classes, instructors needed more than three contacthours each week to fully immerse their students in the active learning experiencesthat were becoming more prevalent.

2) Synchronous Online ToolA synchronous web conferencing tool was viewed as a way to connect

students for out of class group activities, and to connect the instructor withstudents for further exploration of course materials and concepts through problemsolving or exam review sessions. It was also a way to provide a just in time,synchronous learning environment available at any time. This impressive

flexibility made web conferencing an important teaching and learning tool. Withthe added ability to record meeting sessions, a web-conferencing tool couldcombine the synchronous collaborative aspects with the flexibility andconvenience of asynchronous playback allowing participants to review previousmeetings or catch up on missed sessions.

3) Centrally Supported ToolAs often happens on a university campus, pockets of innovators led the

way in new technology implementation. Web conferencing was first introducedto campus in this manner. Several academic departments and research groups hadthe need for an online collaborative tool for their work. Several products wereused across campus, including Skype, WebEx, and Net Meeting. As calls tosupport these disparate tools increased, the central IT organization saw the need tosupport a single web conferencing tool campus wide.

Evaluation Design

Evaluation Development ProcessThe design of this small-scale evaluation followed the principles of a goal-based

model. While there were no published (web or print) goals for the deployment of AdobeConnect, there was an unstated objective to offer those engaged in teaching and learningwith a robust, full-featured, web-conferencing solution that met their needs. There wasalso a desire to offer a product that was easy to use, required no client installation, andwas well-supported among the three major operating systems on campus (Windows, MacOS, and Linux).


10/37

10

The evaluation design used the guiding questions (reviewed earlier) to generate astatus report on actual Adobe Connect usage on campus, to acknowledge areas ofidentified weakness, and to determine an overall satisfaction level of the user base. It wasdetermined early in the planning and design process that a large campus-wide evaluationof instructors was not feasible in the summer. Most faculty members were away from

campus and not available. Initial attempts to contact some faculty members provedunsuccessful. After assessing the reality of limited faculty availability, and an extremelyshort timeline for completing the project, it was decided to design the evaluation as asmall scale effort targeted at a group of fifteen instructors identified as power usersaccording to usage logs generated by the Adobe Connect administrator. Results from thisevaluation should provide a useful foundation for a larger and more thorough campus-wide analysis of Adobe Connect use during the regular academic year.Survey Design

An online user survey in a Google Docs web form was developed to measurespecific interactions and opinions of Adobe Connect. To improve the response rate, the

survey was limited to twenty-three questions, comprised of check box lists, best answer,and open-ended questions. Before sending instructors an email invitation to the surveyform, individual phone conversations were attempted. Four instructors were available fora short phone discussion regarding the intent of the evaluation project and the importanceof their participation. Those unavailable by phone were sent a pre-survey emailexplaining the evaluation effort and how they could help. Instructors were encouraged tocall or email with any questions regarding the evaluation project or survey form in thiscommunication. Email invitations requesting participation in the online survey were sentto all fifteen targeted instructors on July 13. Responses were requested by no later thanJuly 27.

Guided by the evaluation questions, the survey collected information about AdobeConnect usage rates, types of activities conducted, most commonly used tools andfunctions, general satisfaction level, and frustrations or areas of weakness of AdobeConnect. Responses to these questions were expected to allow general patterns to bediscovered from this select user group that would be applicable to the larger campus ofusers.Usage Characteristics

To gauge the extent and variety of web conferencing activity, the first section ofthe survey focused on specific usage details. Users were asked how they used Adobe

Connect in both their teaching and research activities. How did instructors collaboratewith their students? Options included group meetings, office hour sessions, homework orreview sessions, or complete synchronous lectures. The research question sought tounderstand how Adobe Connect helped them conduct their work with remote colleaguesthrough meetings, formal presentations, or industry collaboration. The survey also askedwhere they used Adobe Connect. Knowing where users deploy the tool was consideredimportant information for support personnel who are responsible for maintainingadequate network access and properly equipped computers in a variety of campus


11/37

11

locations including personal offices, departmental conference rooms, shared classroomspaces, and some laboratory environments. This information is also helpful to IT andeLearning staff who are charged with equipping classrooms and other learning spaceswith the technology necessary to conduct an Adobe Connect session with remote users.Users were also asked about the number of attendees in their meetings, including both

local and remote participants. This information is helpful to support staff to betterunderstand the reach of a conference meeting and the associated technical challenges thatcan arise when managing a larger meeting (larger meetings have historically generatedmore complaints to support staff).

Questions related to the specific operating systems helped further define the userbase of the Adobe Connect community. With Flash as the delivery engine, operatingsystem incompatibilities are less frequent, but can still pose challenges for support staff ifolder hardware is being used in a meeting. Ubiquitous mobile devices are increasinglyjoining the web conferencing space. Adobe provides mobile clients for iOS (Apple iPod,iPhone, iPad), Android, and Blackberry devices. Although full functionality with these

devices is not available, participants can generally hear meeting audio, see slides orvideo, and interact with the chat pod.

Teaching and Learning ToolsA series of survey questions was developed to better understand the Adobe

Connect features that are most commonly used by instructors. Learning how audio isused in a meeting was important to better understand the extent of the interaction. As apresenter, the instructor can disable audio from student participants if desired. This canmake managing the meeting easier, but limits collaborative opportunities tochatting. Survey questions about the use of video (webcams), the share pod, chat pod,

poll pod and question pod were all intended to help define the extent of user engagementwith Adobe Connect and the complexity of the meetings being held. An open-endedquestion ended this section of the survey. The question asked for comments regardingAdobe Connects tools and features, and their opinion of it as a collaborative tool. Asone of only two open-ended questions in the survey, this question was viewed as animportant source of detailed information not addressed in the other survey questions. Itwas also expected that it might be used as an opportunity for respondents to vent specificfrustrations (See Appendix A for responses to the open-ended questions).

User FeedbackThe final section of the survey was designed to help answer the guiding questions

of satisfaction level and perceived weaknesses of Adobe Connect. Questions aboutaudio, video, and the share pod were intended to provide detail about technical problemsencountered. Choices of common technical difficulties were provided in each case, but afill in the blank choice was also offered to allow for other problems and to add any detailsthey felt were important. This was followed by the second open-ended question on thesurvey that asked for details about additional problems that they had experienced while


12/37

12

using Adobe Connect. This was intended as a catchall response where participants woulddiscuss any problems they had that were not addressed in earlier survey questions (SeeAppendix A for responses to the open-ended questions).The final questions were used tofurther understand the satisfaction level of users. They asked about the importance oflocal training resources, the usability level of Adobe Connect, and a final overall

impression of the product. Responses here would be helpful in further understanding therelative comfort level of users, the value they placed on support, and a closing impressionregarding satisfaction with the tool.Interview

An interview with Mr. Chad Arney, the Adobe Connect campus administratorwas conducted. This interview was used to gather historical information about MichiganTechs use of web conferencing technology, the process involved in selecting AdobeConnect as the centralized web-conferencing tool, additional details regarding supportneeds and common problems experienced by campus users, and as a source for usagedata generated by reports from the Adobe Connect server. As the campus administrator,

he was also the primary trainer for Adobe Connect users and had important informationabout recurring user problems (both technical and administrative). The seventy-fiveminute interview was conducted on July 21 (See Appendix C for a transcript ofinterview excerpts).

Results and DiscussionSurvey Data Reliability

This was a small-scale evaluation intended for a targeted sample of Adobe

Connect users. The Adobe Connect administrator identified fifteen university instructorsand researchers as good candidates for the evaluation based on their usage rates. Tensurvey recipients had responded by the close of the response window. With such a smallsample population responding, these data are not intended to be representative of theentire campus. They do, however, provide an opportunity to understand some existinguser characteristics, the extent to which the various features of Adobe Connect are used,the frequency of usage, and some of the identified areas of weakness of the applicationamong respondents. A review of this preliminary evaluation data will be beneficialduring the design of any future campus wide evaluation.Complete survey questions and response data are provided in Appendix A.

Forty percent of instructors (Figure 1) said they used Adobe Connect at least oncea week, with the remaining sixty percent indicating usage at least once per month. Asecond introductory question sought information on other web conferencing applicationsthat participants used on campus. Various products were listed, including WebEx,Wimba and Elluminate. All respondents chose other for this question. This seems toindicate a general confusion with the question, so this information was excluded fromfurther analysis.


13/37

13

Usage Data

As a teaching tool, half of the respondents indicated they used Adobe Connect forcollaborative meetings with students, thirty percent used it for online office hours and anadditional thirty percent said they didnt teach with it at all. A similar question regardingresearch activities showed that all respondents used Adobe Connect for collaborativemeetings, sixty percent for formal presentations with professional colleagues, and thirty

percent as a tool to connect with industry partners. Other uses mentioned includedtraining, and connecting with overseas students (Figures 2 & 3).

Ninety percent of survey respondents said they used Adobe Connect in theiroffice, while fifty percent used it in a conference room, at home, or while travelling(Figure 4). Just one person said they used it in a classroom environment. Most users(80%) identified small group meetings of less than ten people as their usual meeting size,while thirty percent also indicated use in groups of up to 25 participants, with a similarnumber also conducting one-to-one sessions. Only twenty percent of users said theyalways recorded their meetings, another twenty percent did so occasionally, and fortypercent never recorded meetings. One person indicated that they recorded presentations

but not meetings, and another was planning to record in the future. The Windowsoperating system was used by eighty percent of users, Mac OS by thirty percent, andLinux by ten percent. These results included Connect users on multiple operatingsystems (Figure 5). The final usage-based survey question revealed that ninety percent ofparticipants did not use the Adobe Connect mobile application, while ten percentindicated use on Apples iOS, and ten percent on Connect for Blackberry.

40%

60%

Figure1Howoftendoyou

currentlyuseAdobeConnect?(N=10)

Useaboutonce

aweek

Useaboutonce

amonth


14/37

14

0

1

2

3

4

5

6

7

meetingswithstudents

onlineof>icehours

homeworksessions

fulllectures supplementalmaterial

other

Figure2-AdobeConnectin

Teaching(N=10)(canrespondtomorethanonechoice)

0

2

4

6

8

10

12

meetingwith

colleagues

formal

presentations

industry

collaboration

other

Figure3-AdobeConnectin

Research(N=10)(canrespondtomorethanonechoice)


15/37

15

Adobe Connect Features

Questions regarding specific features in Adobe Connect were asked to furthermeasure various types of collaboration among survey participants. A critical componentof any web meeting is audio. A meeting host can determine which participants need tosend audio. Respondents reported that sixty percent used audio for all of their meetingparticipants, twenty percent used audio for hosts and presenters only, and twenty percentreported using audio for all participants via a phone conferencing connection. A re-wording of this important question seems in order. From the results as reported, we mustassume that eighty percent of Connect participants used VoIP audio (Voice overInternet). The other twenty percent made sure to note that they used phone lineaudio. The question would have provided more clarity if it had specified VoIP audio, orif there were choices of VoIP audio, phone conferencing audio, or both. Future survey

developers should revisit how to better understand audio use. Half of survey respondentsreported using web-cam videoin all of their meetings, forty percent reported occasionalvideo use, and ten percent never used video.

0

2

4

6

8

10

of>ice conference

room

classroom home when

travelling

Figure4-Wheredoyouuse

AdobeConnect?(N=10)

0

2

4

6

8

10

MacOS Windows Linux AdobeConnect

Mobile

Figure5-ComputerOperatingSystem

UsedwithAdobeConnect(N=10)


16/37

16

The Adobe Connect user interface provides independent pods for its variousfunctions. The share pod is among the most commonly used. From the share pod userscan display presentation slides, a view of the desktop, or a whiteboard for writing. Halfof survey participants reported using the share pod at all their meetings. An additionalforty percent reported occasional share pod use, and only ten percent never used the share

pod. The chat pod, where meeting participants can communicate with each other via textmessaging, was also used by fifty percent of respondents. Another thirty percent reportedoccasional chat pod use and twenty percent never used it. Two less popular pods inAdobe Connect are the question and answer pod, and the poll pod. The question andanswer pod was used occasionally by only twenty percent of participants and seventypercent never used it. The poll pod, used to gauge participant feedback, was only usedoccasionally by ten percent of participants, while eighty percent reported never using thispod. The final question pertaining to Adobe Connect features was an open-endedresponse question. It asked survey participants to give an overall impression of the toolsand features available in Adobe Connect. Not surprisingly, audio difficulties were themost popular response topic here. Three respondents commented on delays and lag with

VoIP audio, with others commenting that conventional phone audio connections were100% better. Several problems reported here related room configuration problems atone or more meeting locations. Issues like low audio, delayed video, or generalconfusion with equipment operation hinted at the need for more thorough training or bestpractices and procedures being offered prior to meetings. See Appendix A for allresponses to the open-ended questions.Adobe Connect Satisfaction Level and Areas for Improvement

A final section of the online survey sought to determine specific problem areas ofAdobe Connect functionality, as identified by the users. Twenty percent of respondents

reported difficulties turning their microphone on or off, ninety percent had problems withaudio feedback, distortion, or sync issues, forty percent reported audio volume controlproblems or audio configuration problems related to the audio setup wizard. One otheruser reported an audio delay problem. Ten percent of users experienced a problemturning their web-cam on or off, ten percent had video distortion problems, and fiftypercent reported problems with video freezing or out of sync with the audio. Othercomments included the connection being too slow when using video, or not relying onvideo. The share pod had some reported difficulties as well, with twenty percentindicating problems uploading slides or displaying properly, twenty percent experiencedslide formatting distortion, twenty percent said desktop sharing did not work properly andten percent had difficulties sharing the whiteboard. Other comments included the report

of a system crash upon ending sharing in a meeting, some file formats not beingrecognized, and the general cumbersome nature of the uploading process.

An open-ended question was provided to allow for further comments. Responsesincluded a report of a generally good usability level, but frustration while trying totroubleshoot audio and video problems. Another user complained of the poor whiteboardperformance and the crude nature of the drawing tools. Please see the complete list ofresponses for this question in Appendix A.


17/37

17

To conclude the survey, sixty percent of respondents said that local support andtraining was very important, and the remaining forty percent said it was somewhatimportant. The usability level of Connect was reported fairly easy to use by thirtypercent of participants, generally usable (with some difficulties) by sixty percent of users,and not intuitive with significant problems by ten percent. Finally, forty percent of users

were very satisfied with their experience with Adobe Connect, forty percent weresomewhat satisfied, and ten percent were somewhat dissatisfied. One user reported theywere very satisfied when it worked, but that some days it was nearly impossible to use.Interview Responses

The interview with Chad Arney, the campus Adobe Connect administrator,revealed some important information about the historical use of web-conferencing at theuniversity, the criteria involved in the selection of Adobe Connect, and his personalexperiences while providing support and training for the product. Some of thisinformation was presented earlier in the background section of the report. (See Appendix

C for a transcript of interview excerpts).

Arney ran report data of usage patterns during the year ending June 30, 2011. Inthat time there were 67 named organizers on campus. Named organizers are people whocan schedule meetings, invite participants and establish participant roles. He estimatedover 300 active users currently on campus, although there was no way to preciselydetermine this number. In the last year there were up to 50 concurrent meetings runningon campus with 434 distinct meeting rooms established and a total of more than 12,200hours of meeting time. The peak meeting connection for the period was the maximumallowable 100 participants for the Adobe-hosted solution. A total of more than 24gigabytes of meeting data were stored on the Adobe servers in the past year.

Arney indicated that there were a wide variety of eLearning activities supportedby Adobe Connect, as reflected in the results of the preliminary survey. Among thesewere group meetings, office hours between students and faculty, student enterprise groupcollaborations with external sponsors or clients, and training efforts to off-campuspartners. He did not personally see as much use of Connect for the delivery ofsynchronous full-length lectures. He also noticed a recent trend on campus for faculty touse web-conferencing to bring working professionals into the classroom to add contextand clarity about the working world that awaits them.

He described some areas where he thought Adobe Connect needed improvement,

including VoIP support, although he said this has vastly improved since 2008. Otherareas included better granularity to permissions settings. He thought customizable roleswould be a great improvement. Reflecting on the role of a support person for web-conferencing software, Arney indicated that remote troubleshooting was the mostdifficult thing to do, because you have no idea about anybodys system, nonewhatsoever. In a nod towards the importance of evaluation in a project or program,Arney said, I would have taken more surveys and taken things more seriously whenpeople complained occasionally. He stressed the importance of paying attention to


18/37

18

formal and informal surveys results. I would have done more data collection during thepilot, I probably would have involved more people in the initial evaluation. Herecommended conducting open forums and training sessions and noted people just wantsomeone to click through the software and see how it works. Summing up his take onthe general satisfaction level of campus users toward Adobe Connect, Arney said he still

hears a few people say they wish they had Marratech, but overall he said I dont hear awhole lot, so in general I think people are really satisfied with it.

Conclusions and Recommendations

Immediate Conclusions This evaluation project helped to develop a better understanding of the usage

characteristics, usage rate, activities conducted, areas of perceived weakness, and

overall impression of the Adobe Connect web conferencing application. Thesedata were obtained from a micro-sample of surveyed users, however, so mindfulinterpretation of findings within this context is important.

One clear result was the importance users placed on campus support andtraining. This sentiment was indicated in the survey results and echoed morestrongly in the interview with the Adobe Connect administrator. It isrecommended that training efforts for campus users are continued and thatconsideration be given to increase both the frequency and variety of thoseofferings. Based on the user data related to audio problems it seems apparent thata more thorough training regimen may reduce the frequency of these

complaints. Particular attention should be paid to the use of the audio setupwizard and applications where a headset with microphone is called for, or where aconference table microphone is a better choice.

It is recommended that support personnel better communicate the appropriateapplications where Adobe Connect can be used effectively. Based on the open-ended responses, it appears that some users are expecting more from the tool thatis reasonable. An example is when it is used with a large group of remote userswho are scattered across multiple countries. Because remote troubleshooting isvery difficult, this user scenario has a high probability for problems and perceivedfailure. If remote users are not trained on the proper use of the tool, the reflex

response may be to blame the technology instead of the users.

IT classroom support personnel should consider improving classroom technologyintegration with Adobe Connect. Based on interview data, there is an increase ininstructors desires to bring remote experts into their classrooms. Improving theaudio and video capabilities of installed computers in selected classrooms forbetter Adobe Connect performance, will help meet this growing need.


19/37

19

Long-Range Planning

The need for a robust web conferencing tool was demonstrated in the evaluationresults. Based on data from the interview with the Adobe Connect administrator,it is recommended that central IT investigate the possibility of hosting the AdobeConnect server on campus instead of the current hosted model that is inuse. Although initial costs for hardware, support personnel, and server licensingwould undoubtedly be higher than current costs, the benefits may be worth theinvestment. A locally hosted solution would allow all university faculty, staff,and students (if deemed necessary) to begin scheduling and hosting their ownAdobe Connect meetings. Fees based on each meeting organizer would go away,as would the current limitations on migrating archived meeting data. Systemhooks into the university learning management system, the university studentinformation system, and other centralized services would allow for tighterintegration with existing services. The option to elevate Adobe Connect from astandalone application to part of a suite of eLearning tools offered by Adobe(along with Adobe Presenter, Adobe Captivate), to expand teaching and learningopportunities would be available.

Results of this evaluation will be useful in the current piloting efforts underway toevaluate and select the next learning management system (LMS) oncampus. Because campus instructors value Adobe Connect for the synchronousconnection it offers them and their students it will be important that any futureLMS supports the integration of Adobe Connect. One scenario would be to allowusers to set up and conduct Adobe Connect meetings from within the LMSitself. Adobe Connect currently offers this functionality through plug-ins,extensions, or third party solutions in many of the new LMS products currentlybeing evaluated on campus.

Because audio difficulties were the most common complaint (both in survey dataand qualitative interview responses) it is recommended that central IT exploreopportunities to provide phone bridging services, provided either in house throughthe existing PBX infrastructure, or through third party providers. Although thisservice is already available externally, an in house solution may prove moreflexible, and allow for campus wide availability. Currently, an Adobe Connectuser needs to make all arrangements for phone conferencing services on theirown. This increases the costs for using Adobe Connect and increases the time it

takes to arrange meetings. This phone conferencing capability could go a longway to solving some of the difficulties often expressed when using VoIP audio.


20/37

20

Evaluation Insights

Due to the time constraints for this evaluation project and the difficulty inconnecting with the evaluation target population in the summer months, this effort wasnecessarily limited in scope and expectations. The small-scale survey did discover

important information regarding how instructors use Adobe Connect, the frequency ofuse, and frustrations using the tool to collaborate with remote students andcolleagues. The survey tool developed to gather this user information may serve as afoundational data collection instrument. To fully understand usage patterns, however, alarger evaluation effort may be needed to more comprehensively engage all campusinstructors and researchers, including those who do not currently use webconferencing. Their comments (regarding lack of use) should help the evaluation teambetter understand the factors that deter some from participating.

Additional face-to-face or phone interviews with a subset of the survey groupwould have been conducted if time had permitted. The unstructured interview with the

Adobe Connect administrator proved very helpful in understanding backgroundinformation on web conferencing on campus, the decision making process that was usedin moving from Marratech to Adobe Connect, and detailed descriptions of user problems,training techniques, and best practices that were developed during the time AdobeConnect has been in use. Similar interviews with users would allow for a customizedexploration of usage patterns, technical problems encountered, and the possible causes ofthose problems. It would also offer detailed information on how they use webconferencing to engage their remote audience. This information could be invaluable toboth support personnel as they troubleshoot user problems and train new users, and forfellow teaching colleagues to compare notes on best practices developed over time.

Future evaluation efforts might also benefit by assessing the student learnerperspective of web conferencing. A second survey instrument aimed at students couldprovide a wealth of information on their experiences in using Adobe Connect, where theyuse it, technical problems encountered (which may be very different from instructorproblems), and the value they assign to the tool as another way to engage with theirinstructors and peers during their studies. An evaluation that measures all parties in aprocess seems more likely to produce well-rounded conclusions and relatedrecommendations.


21/37

21

Appendix A- Adobe Connect Survey Response Report

Appendix A- Adobe Connect Survey Response Report

10responsesSummarySee complete responses

Tell us how often you currently use Adobe Connect

I use Adobe Connect at least once a week. 4 40%

I use Adobe Connect about once a month. 6 60%

I rarely use Adobe Connect. 0 0%

I don't use Adobe Connect. 0 0%

If you use some other web conferencing technology please tell us what it is.

Elluminate 0 0%

Wimba Classroom 0 0%

Go To Meeting 0 0%

Web Ex 0 0%

Google 0 0%

I don't use web conferencing. 0 0%

Other 10 100%

Adobe Connect Usage Details

The following questions will help us better understand the ways you use Adobe Connect in your work.

Tell us how you use Adobe Connect as part of your TEACHING activities.


22/37

22

For collaborative meetings with students. 5 50%

For online office hours with students. 3 30%

To conduct problem solving/homework sessions with students. 1 10%

To conduct fully interactive class lectures 1 10%

To record supplemental materials for on-demand viewing. 0 0%

Other 6 60%

People may select more than one checkbox, so percentages may add up

to more than 100%.

Tell us how you use Adobe Connect as part of your RESEARCH activities.

For collaborative meetings with professional colleagues. 10 100%

For formal presentations with professional colleagues. 6 60%

To collaborate with industry partners. 3 30%

Other 3 30%

People may select more than one checkbox, so percentages may add

up to more than 100%.

Where do you use Adobe Connect?

In my office. 9 90%

In a conference room. 5 50%

In a classroom. 1 10%

At home. 5 50%

When I'm travelling. 5 50%

Other 0 0%

People may select more than one checkbox, so

percentages may add up to more than 100%.

How many participants are in attendance at your Adobe Connect meetings?

more than 25 attendees (large group meeting ) 0 0%

between 10-25 attendees (mid-size meeting) 3 30%

2-9 attendees (small group meeting) 8 80%

one-on-one sessions 3 30%

Other 0 0%




23/37

23

Do you record your meetings and presentations for later playback?

Yes, I always record my meetings and presentations. 2 20%

I occasionally record them. 2 20%

No, I never record them. 4 40%

Other 2 20%

What operating system do you run Adobe Connect on?

Mac OS 3 30%

Windows 8 80%

Linux 1 10%

Adobe Connect Mobile app 0 0%

Other 0 0%



Do you use the Adobe Connect mobile app for Apple iOS, Android, or Blackberry?

I use Adobe Connect Mobile for Apple iOS (iPhone/iPad/iPod). 1 10%

I use Adobe Connect Mobile for Android. 0 0%

I use Adobe Connect Mobile for Blackberry. 1 10%

I don't use the Adobe Connect Mobile app. 9 90%

Other 0 0%


to more than 100%.


Tell us about the tools and features of Adobe Connect that you use regularly.

How is audio used as part of your Adobe Connect meetings?

All meeting participants use audio. 6 60%

Just the hosts and presenters use audio. 2 20%

Other 2 20%


24/37

24

Do you use video (webcams) as part of your Adobe Connect meetings?

I always use video in my meetings. 5 50%

I occasionally use video in my meetings. 4 40%

I never use video in my meetings. 1 10%

Do you use the share pod as part of your Adobe Connect meetings?

I always use the share pod in my meetings. 5 50%

I occasionally use the share pod in my meetings. 4 40%

I never use the share pod in my meetings. 1 10%

What is the share pod? 0 0%

Do you use the chat pod as part of your Adobe Connect meetings?

I always use the chat pod in my meetings. 5 50%

I occasionally use the chat pod in my meetings. 3 30%

I never use the chat pod in my meetings. 2 20%

What is the chat pod? 0 0%

Do you use the question and qnswer (Q&A) pod as part of your Adobe Connect meetings?

I always use the Q&A pod in my meetings. 0 0%

I occasionally use the Q&A pod in my meetings. 2 20%

I never use the Q&A pod in my meetings. 7 70%

What is the Q&A pod? 0 0%

Do you use the poll pod as part of your Adobe Connect meetings?

I frequently use the poll pod in my meetings. 0 0%

I occasionally use the poll pod in my meetings. 1 10%

I never use the poll pod in my meetings. 8 80%

What is the poll pod? 1 10%


25/37

25

What is your overall impression of the tools and features available in Adobe Connect as a collaborative eLearning

tool?

In general its very good. VOIP usually has too much delay to be useful/reliable. Conventional phone connection is 100%

better. In practice, with larger group meetings and with group presentations many times is challenging to get a smoothly

running meeting going. We use it a lot with students in the developing world (as part of the Peace Corps Master's

International program) and we have limited success. Sometimes there is a lot of lag (in audio and with slides changing, video)

which is frustrating. Sometimes the conference room mic is not positioned correctly and the audio is weak. It is diff ...

Adobe Connect Feedback

The following questions involve your experience using Adobe Connect.

Have you experienced any of the following audio problems during your use of Adobe Connect?

problems turning microphone on/off 2 20%

audio feedback, distortion, or sync problems 9 90%

audio volume too loud or too soft 4 40%

problems configuring audio input source with the setup wizard 4 40%

Other 3 30%


to more than 100%.

Have you experienced any of the following video problems during your use Adobe Connect?

problems turning webcam on/off 1 10%

video distortion or quality problems 1 10%

problems configuring webcam during setup 0 0%

video freezes or has sync problems with audio 5 50%

Other 5 50%



Have you had any difficulty using the share pod to share slides, your desktop, or the whiteboard in Adobe

Connect?

slides would not upload or display properly 2 20%

slide formatting was distorted 2 20%

desktop sharing did not work properly 2 20%

whiteboard sharing did not work properly 1 10%

Other 7 70%




26/37

26

Please tell us any additional information regarding problems you have experienced when using Adobe

Connect?

I used it for about 30 class sessions last semester, On two occasions a DL student connecting from Merc Marine had

trouble joining the session. See response above for specific problems. I would like to clarify for the questions below.

While I do think the usability level of adobe connect is generally good, troubleshooting issues with audio and video can

be extremely frustrating - for important collaboration meetings we will invite Chad Arney to attend if possible and he can

help us if anyone has difficulties. See the text above. None none See notes above. The tool's weakest point

is the white-bo ...

Do you think it is important to have local support and training resources available to Adobe

Connect users on campus?

Very important 6 60%

Somewhat important 4 40%

Not very important 0 0%

Other 0 0%

How do you perceive the usability level of Adobe Connect?

Fairly straightforward and easy to use. 3 30%

Some difficulties with features, but generally useable. 6 60%

Significant difficulties with features and not intuitive to use. 1 10%

Other 0 0%

What is your overall impression of using Adobe Connect as a web conferencing tool for teaching and learning?

I am very satisfied with my experience. 4 40%

I am somewhat satisfied with my experience. 4 40%

I am somewhat dissatisfied with my experience. 1 10%

I am very dissatisfied with my experience. 0 0%

Other 1 10%

Thank you for taking the time to repsond to our Adobe Connect User Survey.

Number of daily responses


27/37

27

Appendix A- Adobe Connect Survey Response Report (open-ended survey questionssummary)

Question Responses

What is your overall impression ofthe tools and features available in

Adobe Connect as a collaborative

eLearning tool?

- In general its very good. VOIP usually has too muchdelay to be useful/reliable. Conventional phone connectionis 100% better.

- In practice, with larger group meetings and with grouppresentations many times is challenging to get a smoothly

running meeting going. We use it a lot with students in the

developing world (as part of the Peace Corps Master's

International program) and we have limited success.

Sometimes there is a lot of lag (in audio and with slides

changing, video), which is frustrating. Sometimes the

conference room mic is not positioned correctly and the

audio is weak. It is difficult to get all the

attendants/participants to understand the need for a headset;

so many times they don't really participate or just use thechat box, which is not very effective for groupcollaboration. I also think part of the difficulty lies in the

fact that web conferencing 'culture' is new and many people

feel uncomfortable if they are not present physically in the

room (they can't feel the non-verbal climate in the room).

Also, those presenting may not see the need to use (or are

simply not used to using) the connect pro tools (arrow for

pointing, the whiteboard, etc.) so those who are on-line

don't see what the speaker is talking about. That said, I do

use it frequently for one-on-one meetings with a US

university collaborator (who is used to several different webconferencing systems) and it works great.

- I have some trouble almost every time I use it - either on

my end or with the participants. Explaining to a participant

how to switch between walkie-talkie and hand-free is very

tiresome and time consuming. For small groups, managing

a problem is not a big issue. For larger groups (5+) it's very

frustrating. At one time I used it in the classroom. I've given

up on that because I didn't have the time to shepherd

participants through their issues.

-It has served us well, except for the difficulties with the

audio portion.

-It is a very useful tool, but I dont use most of thefeatures.

-Overall, the tools work well for the work that I need to

do. Occasionally, however, there seems to be a delay in the

audio/video (causing people to talk over one another), and

sometimes the system crashes when the person sharing tries

to quit sharing. These problems are inconvenient for the

group. The system would be much better if these issuescould be resolved, but using Adobe is much more effective


28/37

28

and efficient than traveling to meet with research

colleagues.

-The tool and the features are good. My only problem is

the limited Linux support.

It is getting better, but it is impossible to carry a

conversation under Linux unless both parties put on ear-phones. Connect add in for Firefox/Linux is a must. This is

one of few remaining reasons why I keep a windows

bootable machine and it is a pain to boot/reboot and

sometimes access documents which are stored in the Linux

partition.

- This works great for meeting weekly or so with our

Michigan Tech partners in Ann Arbor. It sure makes theresearch project go smoother to be able to share files from

all of us.

- I haven't delved into them much yet. The software issurely sophisticated, but not always intuitive.

-It seems to be a good tool (and getting better) for

collaborating with the high school teachers that are partners

in our program. They are able to use it (mostly) without

many technical problems. Since I communicate with

roughly a dozen teachers at a time, most of thecommunication is "one way" from me to them. The chat

pod is used to allow the teachers to send information, such

as answers to a question or general responses to the dialog.

Video - I usually broadcast video of me, but I'm not surehow important this is in my meetings.

Audio - this is critical we can't hold the meetings without it.

The latest version of AC seems to be less problematic.

My main concern is that the remote users have an easy timeconnecting, getting audio and participating in the meetings.

Please tell us any additional

information regarding problems

you have experienced when using

Adobe Connect

- I used it for about 30 class sessions last semester, On two

occasions a DL student connecting from Merc Marine had

trouble joining the session.

- The tool's weakest point is the white-board. The drawing

tool is crude at best and difficult to use.

- The desktop sharing menus sometimes goes away,

probably a function of the computer in DOW 875.

- Too much latency with connections to students abroad.


29/37

29

Appendix B- Adobe Connect Survey Questions

Adobe Connect User Survey

This survey is designed to learn more about Adobe Connect web conferencing users at Michigan Tech. Thank you for taking some time to help us better

understand how this important collaborative tool is being used.

Tell us how often you currently use Adobe Connect *

Includes all users including those who manage, host, or particpate in Adobe Connect meetings.

I use Adobe Connect at least once a week.

I use Adobe Connect about once a month.

I rarely use Adobe Connect.

I don't use Adobe Connect.

Page 2

Note: "Go to page" selections will override this navigation. Learn more.

After page 1 Continue to next page

If you use some other web conferencing technology please tell us what it is.

Elluminate

Wimba Classroom

Go To Meeting

Web Ex

Google

I don't use web conferencing.

Other:

Page 3

Note: "Go to page" selections will override this navigation. Learn more.

After page 2 Continue to next page

Adobe Connect Usage Details

The following questions will help us better understand the ways you use Adobe Connect in your work.

Tell us how you use Adobe Connect as part of your TEACHING activities. *

The course can be face-to-face, blended, or fully online.

For collaborative meetings with students.

For online office hours with students.

To conduct problem solving/homework sessions with students.

To conduct fully interactive class lectures

To record supplemental materials for on-demand viewing.

Other:

Tell us how you use Adobe Connect as part of your RESEARCH activities. *

For collaborative meetings with professional colleagues.

For formal presentations with professional colleagues.

To collaborate with industry partners.

Other:


30/37

30

Where do you use Adobe Connect? *

check all that apply

In my office.

In a conference room.

In a classroom.

At home.

When I'm travelling.

Other:

How many participants are in attendance at your Adobe Connect meetings? *

Please select all that apply

more than 25 attendees (large group meeting )

between 10-25 attendees (mid-size meeting)

2-9 attendees (small group meeting)

one-on-one sessions

Other:

Do you record your meetings and presentations for later playback? *

Yes, I always record my meetings and presentations.

I occasionally record them.

No, I never record them.

Other:

What operating system do you run Adobe Connect on? *


Mac OS

Windows

Linux

Adobe Connect Mobile app

Other:

Do you use the Adobe Connect mobile app for Apple iOS, Android, or Blackberry? *

The Adobe Connect Mobile app allows you to participate in meetings via your mobile device.

I use Adobe Connect Mobile for Apple iOS (iPhone/iPad/iPod).

I use Adobe Connect Mobile for Android.

I use Adobe Connect Mobile for Blackberry.

I don't use the Adobe Connect Mobile app.

Other:


Tell us about the tools and features of Adobe Connect that you use regularly.

How is audio used as part of your Adobe Connect meetings? *

Do only the hosts and presenters use audio, or all participants?

All meeting participants use audio.

Just the hosts and presenters use audio.

Other:


31/37

31

Do you use video (webcams) as part of your Adobe Connect meetings? *

I always use video in my meetings.

I occasionally use video in my meetings.

I never use video in my meetings.

Do you use the share pod as part of your Adobe Connect meetings? *

The share pod allows hosts and presenters to share slide presentations, their computer desktop, or the whiteboard

I always use the share pod in my meetings.

I occasionally use the share pod in my meetings.

I never use the share pod in my meetings.

What is the share pod?

Do you use the chat pod as part of your Adobe Connect meetings? *

The chat pod allows meeting participants to type messages to the entire group or selected members of the group

I always use the chat pod in my meetings.

I occasionally use the chat pod in my meetings.

I never use the chat pod in my meetings.

What is the chat pod?

Do you use the question and qnswer (Q&A) pod as part of your Adobe Connect meetings? *

The Q&A pod allows meeting participants to post questions to the group.

I always use the Q&A pod in my meetings.

I occasionally use the Q&A pod in my meetings.

I never use the Q&A pod in my meetings.What is the Q&A pod?

Do you use the poll pod as part of your Adobe Connect meetings? *

The poll pod allows hosts to post poll questions to participants to gague opinions or levels of understanding.

I frequently use the poll pod in my meetings.

I occasionally use the poll pod in my meetings.

I never use the poll pod in my meetings.

What is the poll pod?

What is your overall impression of the tools and features available in Adobe Connect as a collaborative eLearning tool? *

Adobe Connect Feedback

The following questions involve your experience using Adobe Connect.

Have you experienced any of the following audio problems during your use of Adobe Connect? *


problems turning microphone on/off

audio feedback, distortion, or sync problems

audio volume too loud or too soft

problems configuring audio input source with the setup wizard

Other:

Have you experienced any of the following video problems during your use Adobe Connect? *


problems turning webcam on/off

video distortion or quality problems

problems configuring webcam during setup

video freezes or has sync problems with audio

Other:


32/37

32

Have you had any difficulty using the share pod to share slides, your desktop, or the whiteboard in Adobe Connect? *


slides would not upload or display properly

slide formatting was distorted

desktop sharing did not work properly

whiteboard sharing did not work properly

Other:

Please tell us any additional information regarding problems you have experienced when using Adobe Connect? *

Do you think it is important to have local support and training resources available to Adobe Connect users on campus? *

(In addition to the extensive online training resources available online from Adobe)

Very important

Somewhat important

Not very important

Other:

How do you perceive the usability level of Adobe Connect? *

Is it relatively intuitive to operate, given a reasonable amount of time to prepare and train?Fairly straightforward and easy to use.

Some difficulties with features, but generally useable.

Significant difficulties with features and not intuitive to use.

Other:

What is your overall impression of using Adobe Connect as a web conferencing tool for teaching and learning? *

I am very satisfied with my experience.

I am somewhat satisfied with my experience.

I am somewhat dissatisfied with my experience.

I am very dissatisfied with my experience.

Other:

Page 4 After page 3 Continue to next page

Thank you for taking the time to repsond to our Adobe Connect User Survey.


33/37

33

Appendix C- Selected Responses from Chad Arney Interview (Adobe Connect SystemAdministrator) on July 21, 2011

Q: Can you provide timeline information regarding the switch to Adobe Connect at

MTU?

A: We began a pilot of Adobe Connect in August of 2008. Marratech (previous webconferencing product) kept running during Fall 2008. We stopped Marratech support inspring 2009. We turned off the Marratech server in summer 2009.

Q: What other web conferencing products did you consider besides Adobe Connect?

A: We looked at Elluminate, Wimba, AT&T Connect, in July of 2008.

Q: What were some of the advantages of Adobe Connect that stood out at you?

A: Adobe Connect was Flash-based, so it supported the three main platforms (Windows,Mac, Linux). The parity of the user experience across platforms was really important.

A: The look and feel of Adobe Connect was really important and it really seemed to fitwith a higher education institution, versus the corporate feel of WebEx and GoToMeeting.

A: The fact that we could get into it (Adobe Connect) really inexpensively really made abig difference.

A: This was a real low risk pilot (Adobe Connect). We could get started for less than$1,000, and we only had an obligation for a year.

A: Price was a factor, function, form, technology, and scalability were all factors, andAdobe Connect really just struck everybody as being the easiest to use interface, the mostcompatible...

A: That was a huge deciding factor, was the fact that we didnt have to deal with otherinstitutions network policies for our online learners, or research colleagues, or enterprisesponsors, or corporate partners. We didnt have to say: yah, if you want to meet with ususing the great new synchronous online web conferencing tool youre going to have topunch a hole in your firewall.

Q: Can you provide some basic statistics concerning usage of Adobe Connect in the pastyear?

A: Currently 67 named organizers (create, manage, and design meetings). There areapproximately 300-350 active users, but theres no way to determine this number


34/37

34

precisely. We have had up to 50 concurrent meetings happening on campus. Weve hadover 12,000 meeting hours this year at MTU.

Q: Why did MTU choose the Adobe-hosted solution for the product?

A: The scalability of licensing, inexpensive, no hardware to support.The downside included an inability to integrate the hosted Adobe Connect with currentcampus enterprise applications. There is also no way to move all of the archived meetingdata to an on-site deployment. This policy might change.

Q: What are some of the weaknesses of Adobe Connect that you have discovered?

A: We should have been a little more critical of the VoIP implementation in version 7 ofAdobe Connect. The best I can tell they have resolved many of the VoIP issues inversion 8, but I think we should have been a little more critical of that at the time.

A: The permissions and granularity of permissions needs to be more robust. Audiomanagement. The ability of a host to know more about the computer setup ofparticipants in a meeting. Remote troubleshooting is probably the hardest thing aboutany web-conferencing software, because you have no idea about anybodys system, nonewhatsoever.

Q: Can you give us some lessons learned you discovered through this process?

A: I would have taken more surveys and taken things more seriously, when peoplecomplained.

A: Pay attention to informal and formal survey results. I would have done more datacollection during the pilot. I probably would have involved more people in the initialevaluation...identifying the stakeholders.

A: I recommend open forums and training. I got a lot of good feedback from that.

A: I would do more communication to campus... people just want to see what this lookslike. People want someone to click through the software and see how it works.

Q: Are users happy with Adobe Connect?

A: I still hear a few people say they wish they had Marratech. I dont hear a whole lot,so in general I think people are really satisfied with it.


35/37

35

Appendix D-Adobe Connect Server Usage Report 2010-11 (summary page)

My Profile Help Logout: Jeff Toorongian

Manager Reports

Reporting database synchronized at a regular interval. Report data may be delayed by up to one day.

Home Content Meetings Reports Administration Search ...


36/37

36

Appendix E-Evaluators Program Description (EPD):A Small-Scale Evaluation of Adobe Connect as a University

+RZH[WHQVLYHLV$GREH

&RQQHFWXVHIRUZHE

FRQIHUHQFLQJDFWLYLWLHVDPRQJ

LQVWUXFWRUV"

,QWURGXFH$GREH&RQQHFW

DVWKHFDPSXVVXSSRUWHG

ZHEFRQIHUHQFLQJ

WHFKQRORJ\LQ

6XUYH\

UHVSRQVHV

LQWHUYLHZ

UHVSRQVHV

RQOLQHVXUYH\

LQWHUYLHZV\VWHP

XVDJHORJV

FDPSXVLQVWUXFWRUV

RIUHFRUG

$GREH

&RQQHFW

DGPLQLQVWUDWRUWUDLQH

RQOLQHVXUYH\WRRO

LQ*RRJOH'RFV

LQWHUYLHZ

'

HVFULSWLYH

$

QDO\VLV

(YDOXDWRUWRDQDO\]H

DQG$GREH&RQQHFW

DGPLQLVWUDWRUWUDLQHU

WRSURYLGHV\VWHP

ORJV

H/HDUQLQJD

QG,7

VWDII

$GREH

&RQQHFWXVHUV

VWXGHQWFHQ

WUDO

DGPLQLVWUDWLRQ

:KDWWHDFKLQJDQGOHDUQLQJ

DFWLYLWLHVDUHFRQGXFWHGZLWK

$GREH&RQQHFWPHHWLQJV

SUREOHPVROXWLRQ

RIILFH

KRXUV

VRIWZDUH

GHPRQVWUDWLRQ

HWF

"

7RSURYLGHLQVWUXFWRUV

ZLWKDIXOOIHDWXUHG

ZHE

EDVHGFRQIHUHQFLQJWRRO

WKDWRIIHUHGIOH[LELOLW\WR

VXSSRUWGLYHUVHWHDFKLQJ

DQGOHDUQLQJSHGDJRJLHV

6XUYH\

UHVSRQVHV

LQWHUYLHZ

UHVSRQVHV

RQOLQHVXUYH\

LQWHUYLHZ

FDPSXVLQVWUXFWRUV

RIUHFRUG

$GREH

&RQQHFW


RQOLQHVXUYH\WRRO

LQ*RRJOH'RFV

LQWHUYLHZ

'

HVFULSWLYH

$

QDO\VLV

(YDOXDWRU

H/HDUQLQJD

QG,7

VWDII

$GREH

&RQQHFWXVHUV

VWXGHQWFHQ

WUDO

DGPLQLVWUDWLRQ

:KDWLVWKHVDWLVIDFWLRQOHYHO

RI$GREH&RQQHFWDVWKH

FDPSXVVXSSRUWHGZHE

FRQIHUHQFLQJWRRO"

7RSURYLGHLQVWUXFWRUZLWK

DZHEFRQIHUHQFLQJWRRO

WKDWZLOOEHHDV\WRXVH

DQGDOORZLQVWUXFWRUVWR

WDNHFKDUJHRIWKHLUZHE

FRQIHUHQFLQJQHHGV

6XUYH\

UHVSRQVHV

LQWHUYLHZ

UHVSRQVHV

RQOLQHVXUYH\

LQWHUYLHZ

FDPSXVLQVWUXFWRUV

RIUHFRUG

$GREH

&RQQHFW


RQOLQHVXUYH\WRRO

LQ*RRJOH'RFV

LQWHUYLHZ

'

HVFULSWLYH

$

QDO\VLV

(YDOXDWRU

H/HDUQLQJD

QG,7

VWDII

$GREH

&RQQHFWXVHUV

VWXGHQWFHQ

WUDO

DGPLQLVWUDWLRQ

:KDWDGGLWLRQDOIXQFWLRQDOLW\

RUDUHDVRIZHDNQHVVLQ

$GREH&RQQHFWDUHLGHQWLILHG

E\WKHXVHUFRPPXQLW\"

7RSURYLGHDOHDGLQJ

ZHEFRQIHUHQFLQJWRRO

WKDWSURYLGHVWKH

IXQFWLRQDOLW\GHPDQGHG

E\LQVWUXFWRUV

DQGLVZHOO

VXSSRUWHGDQG

FRQWLQXRXVO\XSGDWHGE\

WKHFRPSDQ\

6XUYH\

UHVSRQVHV

LQWHUYLHZ

UHVSRQVHV

RQOLQHVXUYH\

LQWHUYLHZ

FDPSXVLQVWUXFWRUV

RIUHFRUG

$GREH

&RQQHFW


RQOLQHVXUYH\WRRO

LQ*RRJOH'RFV

LQWHUYLHZ

'

HVFULSWLYH

$

QDO\VLV

(YDOXDWRU

H/HDUQLQJD

QG,7

VWDII

$GREH

&RQQHFWXVHUV

VWXGHQWFHQ

WUDO

DGPLQLVWUDWLRQ

(YDOXDWLRQ4XHVWLRQ

3URJUDP2EMHFWLYH*RDOV

$FWLYLW\LHVWR

2EVHUYH

'DWD6RXUFH

3RSXODWLRQ6DPSOH

'HVLJQ

'

DWD$QDO\VLV

5HVSRQVLELOLW\

$XGLHQFH


37/37

Appendix F-Evaluation Project Timeline

0RQ 6XEPLW(3'7LPHOLQH3URMHFWGHVFULSWLRQ

7XHV %HJLQZULWLQJVXUYH\TXHVWLRQV

:HG &RQWLQXHVXUYH\TXHVWLRQVFROODERUDWHZLWK&KDGDERXWKLVWRULFDO$GREH&RQQHFWSURMHFWUROORXWGHWDLOV

7KXUV &RQWLQXHVXUYH\TXHVWLRQVFRQILUPSURMHFWDSSURYDOIURP3URI3HUNLQV

)UL)LQDOL]HVXUYH\TXHVWLRQVVFKHGXOHLQWHUYLHZZLWK$GREH&RQQHFWDGPLQLVWUDWRUWUDLQHU5HFHLYHGIHHGEDFNDQGDSSURYDOIURP3URI3HUNLQVWRGD\

6DW %HJLQVXUYH\LQVWUXPHQWHGLWLQJLQ*RRJOH'RFV

6XQ &RQWLQXHVXUYH\HGLWLQJLQ*RRJOH'RFVSODQIRUGHSOR\PHQWRQ6DW-XO\

7XHV

:HG

7KXUV

)UL )LQDOL]HRQOLQHVXUYH\LQFOXGLQJIHHGEDFNIURP$GREH&RQQHFWDGPLQWUDLQHU

6DW&RQILUPILQDOVXUYH\VWDWXVDQGVHQGRXWVXUYH\LQYLWDWLRQZHHNUHVSRQVHZLQGRZHPDLOWRLQVWUXFWRUVDQG$GREH&RQQHFWOLVWPHPEHUV

6XQ 0RQLWRURQOLQHVXUYH\IRUUHVSRQVHV

0RQ 'UDIWLQWHUYLHZTXHVWLRQVIRU&KDG$UQH\LQWHUYLHZ

7XHV %HJLQDQDO\VLVRILQLWLDOVXUYH\GDWDLIDQ\

:HG 0RQLWRURQOLQHVXUYH\IRUUHVSRQVHV

7KXUV 0RQLWRURQOLQHVXUYH\IRUUHVSRQVHV

)UL

6DW 0RQLWRURQOLQHVXUYH\IRUUHVSRQVHV

6XQ 6WDUWGUDIWRIILQDOUHSRUW3XUSRVH%DFNJURXQG

0RQ &RQWLQXHGUDIWRIILQDOUHSRUW3XUSRVH%DFNJURXQG

7XHV &RQWLQXHGUDIWRIILQDOUHSRUW'HVLJQGHVFULSWLRQ5HVXOWV

:HG2QOLQHVXUYH\UHVSRQVHVGXH%HJLQFRPSLOLQJVXUYH\UHVSRQVHV&RQWLQXHGUDIWRIILQDOUHSRUW'HVLJQGHVFULSWLRQ5HVXOWV

7KXUV,QWHUYLHZZLWK$GREH&RQQHFWDGPLQLVWUDWRU&KDG$UQH\$QDO\]HUHVSRQVHGDWDEHJLQGHYHORSLQJGDWDFKDUWVJUDSKVDVQHHGHG&RQWLQXHGUDIWRIILQDOUHSRUW5HVXOWV'LVFXVVLRQ

)UL 7UDQVFULEH&KDG$UQH\LQWHUYLHZIIURPUHFRUGLQJV&RQWLQXHGUDIWRIILQDOUHSRUW5HVXOWV'LVFXVVLRQ

6DW %HJLQZULWLQJILQDOHYDOXDWLRQUHSRUW3XUSRVH

6XQ %HJLQZULWLQJILQDOHYDOXDWLRQUHSRUW%DFNJURXQG,QIR

0RQ FRQWLQXHHYDOXDWLRQUHSRUWZULWLQJ

7XHV2QOLQHVXUYH\UHVSRQVHVGXH%HJLQFRPSLOLQJVXUYH\UHVSRQVHV&RQWLQXHGUDIWRIILQDOUHSRUW'HVLJQGHVFULSWLRQ5HVXOWV

:HG)LQDOFKHFNIRUDQ\ODWHVXUYH\UHVSRQVHV8SGDWHGDWDFKDUWVJUDSKVIRUDQ\ODWHUHVSRQVHV&RPSOHWH(YDOXDWLRQ'HVLJQVHFWLRQ

7KXUV &RQWLQXHHYDOXDWLRQUHSRUWZULWLQJDUUDQJHIRUSURRIUHDGHU

'$7( '$< 7$6.V

)UL0DNHJUDSKVIURPVXUYH\GDWDIRUXVHLQERG\RIUHSRUW&RQWLQXHHYDOXDWLRQUHSRUWZULWLQJVXEPLWGUDIWWRSURRIUHDGHUIRUFRPPHQWV

6DW )LQDOUHYLHZRIUHSRUWUHFHLYHFRPPHQWVIURPUHYLHZHU'UDIWH[HFXWLYHVXPPDU\DQGUHIOHFWLRQVHFWLRQV

6XQ &RPSOHWH([HFXWLYHVXPPDU\/HDUQLQJ5HIOHFWLRQDQGDSSHQGLFHVVHFWLRQV3URRIUHDG

0RQ )LQDO(YDOXDWLRQ3URMHFW'XH

'$7( '$< 7$6.V

a small-scale evaluation of adobe connect as a university web-conferencing tool

Documents