rsinstructionaldesign.comrsinstructionaldesign.com/ist 622/solano ist622 project.docx · web...

Running head: IRRIGATION CONTROLLER TRAINING 1

Evaluation of the Irrigation Controller Training Module

Royins Solano

California State University – Monterey Bay

IST622 Assessment and Evaluation

Dr. Bude Su

July 25, 2017

IRRIGATION CONTROLLER TRAINING 2

TABLE OF CONTENTS

Introduction......................................................................................................................................3

Methodology..................................................................................................................................3

Prototype......................................................................................................................................3

Learners........................................................................................................................................5

Tryout conditions.........................................................................................................................6

Tryout process..............................................................................................................................6

Results..............................................................................................................................................9

Entry conditions...........................................................................................................................9

Instruction....................................................................................................................................9

Outcomes.....................................................................................................................................9

Recommendations......................................................................................................................11

Summary........................................................................................................................................13

Appendices....................................................................................................................................20

.


Section I

Introduction

The Irrigation Controller training module was created with the objective of training

employees of a landscaping business on the basics of programming an irrigation controller. The

target audience have experience working on landscaping but not on irrigation. Approximately 28

employees fit this description. Currently the business does not have a process for training

employees on irrigation. Employees must learn the basic concepts of irrigation in order to

program a controller. Five learners participated in the evaluation of the training module.

For this tryout, the first module on “How to program an irrigation controller” was used.

The tryout measures the usability of the module, learning gains resulting from the instruction,

and users’ reactions through observations, pre- and post-tests, and a questionnaire. The tryout

will also help the designer understand areas needing improvement in terms of design, content,

and flow of the instruction.

Section II

Methodology

Prototype

The prototype used for this evaluation is the first part of my capstone project in “How to

program an irrigation controller”. It is focused on basic concepts and process. This training

module consists of four topics: Irrigation Controller Benefits, Types, Features and Basic Program


Sequence. The prototype module for this evaluation teaches how to program a popular residential

controller. The X-Core by Hunter.

The prototype is accessible through computer and mobile devices – laptop, tablet, and

smartphone. The prototype was built using Adobe Captivate and the learners accessed the

training through an online link. The duration of the training was approximately ten to fifteen

minutes. The module has audio instructions and provides the learners the option to read or to

listen to audio about each topic. There is opportunity for optional practice to help learners engage

with the course. At the end of the training, the learners complete a 5 question quiz. They receive

immediate feedback based on the chosen answer and a final pass or fail score. If they pass, they

are presented with a certificate of completion, otherwise they have infinite attempts to try again.

The prototype was designed with the intention that learners could get a first approach to

irrigation controllers without the fear of breaking something. Some pictures show actual jobs

completed by the company. See examples of the prototype slides in tables 2.1 and 2.2.


Table 2.1


Table 2.2

Learners

The learners were meant to represent the wide range of demographics for this

landscaping business. Employees share gender, language and occupation. Their age goes from 21

to 51 years old. They are not provided training in irrigation. Some of them claim having seen

work done on irrigation but have no experience of their own. The average age for the learners

was thirty-five.

Forty percent of the learners had some type of knowledge about irrigation. Helping to

replace a broken sprinkler was the most common interaction. Sixty percent of the learners had no

knowledge in irrigation. See table 2.3


Experience

Irrigation Experience No Irrigation Experience

Table 2.3

High school is the highest education level achieved by the learners. They can read and

write but some senior employees have trouble with the computer interaction.

Tryout Conditions

The tryout conditions were designed to replicate the training environment of the

employees. Learners took the training independently on a windows computer with Internet

connection in the main business office. The designer stood up by their side to take notes for the

usability test and help them with any inquiry. He guided the learners through the correct order to

take the survey, tests, content and questionnaire. The learners did not repeat any part of the

training except for the quiz questions to evaluate learning effectiveness.

Tryout Process

The tryout process was comprised of six parts: survey, pre-test, training, post-test,

questionnaire, and observation. The survey and questionnaires were created as a word document.


Then administered verbally in person. All information was confidential and anonymous.

Learners provided their names in the learning project but it was not recorded for reporting in

order to maintain confidentiality. Personal information only recorded demographics. The five

learners were required to complete each of the below items.

1. Survey

The learners responded to the survey first. The survey captured the demographic

information and any prior experience with irrigation. Learners were asked to provide their

occupation, gender, ethnicity, and age. They were asked if they had previous experience in

irrigation. The ones who said they had experience, explained they are rarely sent to help with

irrigation problems and clarified they have helped to replace a sprinkler but not yet to program a

controller. See appendix 5.1.

2. Pre-Test

The pre-test was an integral part of the evaluation process since this captured the learner’s

knowledge and skill set prior to taking the training. The questions were designed to assess the

most basic concepts of irrigation and the sequence to program a controller. The questions

specifically addressed aspects that were taught in the training. For example, the benefits and

features of irrigation controllers. These questions were the same in the training module’s pre-test

and final evaluation. See appendix 5.2.

3. Training

The learners accessed the training through the link provided. They participated in the

training immediately after taking the pre-test. The web-based training was approximately ten

minutes in duration. You may access the training link in appendix 5.3.


4. Post-Test

Once the learners finished the training, they were instructed to continue taking the post-

test. The post-test questions were identical to the pre-test. Since the evaluation was assessing

knowledge before and after the training intervention, identical pre- and post-tests would be

appropriate for this evaluation. The post-test was also an essential part of the tryout process. In

order to measure any change in knowledge, I compared the scores of the pre-test with the post-

test results. See appendix 5.2.

5. Questionnaire

The questionnaire was designed to capture the comments, reactions, and perceptions of

the learners. Most of the answers to the questions included a Likert scale. For example, one

question read: “How confident do you feel in programming an irrigation controller?” The range

of answers included: very confident, confident, undecided, not confident, and not confident at all.

Questions addressed the level of difficulty/ease in following the module instructions, the level of

difficulty/ease of the quizzes, and any level of frustration the learners may have experienced.

There were two open-ended questions that asked the learners to record anything they liked or

disliked about the training. See an example of the questionnaire in Appendix 5.4.

6. Observation

Five learners were observed individually in-person. The learners were contacted and

scheduled for observation during June. I approached the learners the day before the scheduled

observation to remind them of our session. The survey was in person and I provided access to the

tests and training. All learners accessed the tests and training from an office computer. I stood


near them and observed them taking the training on the computer. I created a checklist that I used

for the observations. See the observation checklist in Appendix 5.5.

Section III

Results

Entry Conditions

The intended and observed entry conditions were not the same. Not all learners had

enough computer experience to feel comfortable with the mouse. I expected none of them to

have experience in irrigation because they are dedicated to landscaping, yet forty percent of the

learners said they had some type of knowledge about irrigation. Unfortunately, they did not pass

the pre-test. One of the learners faced a challenge using the mouse, maybe he is not a regular

computer user and felt cumbersome. He was among the senior employees so maybe age played a

factor here.

Instruction

The observed instruction was consistent with the intended instruction. Learners who

experienced trouble with the instruction were timely assisted. The learners followed the module

instruction, interacted with the practice, and completed the final evaluation. The most common

feedback from the learners was that the instruction provided very detailed information they could

not easily find such as the programming sequence. Most learners felt confident in responding to

the required tasks. The training navigation was intuitive for all but one of the observed learners,

he mentioned he got confused with the navigation bar.

Outcomes


Before I conducted the tests, I hypothesized that the training would improve the learner’s

knowledge in irrigation as reflected through higher test scores. The null hypothesis was that there

would be no statically significant difference in the pre and post test scores. I ran the paired two

sample t-test for dependent samples. The degree of freedom was 4. I used one-tail results for a

directional hypothesis. The t-test results are below in table 3.11. Since the test statistic: 13.88

was higher than the one-tailed critical value of 2.13, I reject the null hypothesis, and accept the

research hypothesis that there is in fact statistical difference. The training was a significant factor

in the learners’ improved post-test scores.

t-Test: Paired Two Sample for Means

Pre-Test Post-TestMean 1.2 4.6Variance 0.7 0.3Observations 5 5Pearson Correlation 0.763762616Hypothesized Mean Difference 0df 4t Stat -13.88044188P(T<=t) one-tail 7.80959E-05t Critical one-tail 2.131846786P(T<=t) two-tail 0.000156192t Critical two-tail 2.776445105

Table 3.11

Pre and post-test results (See Summary) show that learners achieved significant learning

gains as a result of completing the module.

While the learners had varying degrees of experience on irrigation – some said they know

how to replace a sprinkler – all learners experienced an improvement in their knowledge of


irrigation and irrigation controllers. Overall, learners experienced the most improvement in

understanding when can residential and commercial controllers be used.

Recommendations

Based on the evaluation results, the following recommendations are provided:

1) Provide clearer instruction to navigate the course using the buttons. I assumed all

employees would be fine using a computer and a mouse but some of them had trouble

navigating the course. I will add more detailed instructions near the first slides of the

presentation.

2) Include more practice items. The current module only has one practice item in addition to

the lesson interactions of hovering the mouse over controller buttons to display the

description of each feature. It is recommended to have two practice items per topic to

allow the learners to get familiar with the content before getting to the quiz.

3) Motivate the learners. Not all employees seem excited about learning. Some of them

were avoiding the opportunity with excuses. For my capstone evaluation, I will try to

motivate them mentioning some other extra benefits of having this knowledge and make

it more appealing.

4) Include more test items. It would be good to include more test items so that learners can

have better chance to pass. With this version, the matching questions test for more than

one concept at the time. It could be good to include another question for each one of

those, so that participants can get partial credit if they don’t get the first one right.

Section IV


Summary

The data was obtained through observation. I entered the scores from the pre- and post-test into

an Excel spreadsheet. The chart below demonstrates the scores for the pre- and post-tests. There

were five questions in total for the pre- and post- tests. All five of the learners scored higher

during the post-test. The average score for the pre-test was 1.2 or 24 percent. The average score

for the post-test was 4.6 or 92 percent. See table 4.1 for the distribution of the test scores. The

chart in table 4.2 demonstrates the difference between the pre- and post-test scores.

Pre-test

Scores

Post-test

Scores

2 51 42 50 41 5

Table 4.1

1 2 3 4 50

1

2

3

4

5

6

Learning Effectiveness

Pre-test Scores Post-test Scores


Table 4.2

Learners experienced the greatest improvement in their scores with questions 2, 4 and 5.

Question 1: Order the steps in the correct sequence:

a. Set the current time and date

b. Set start times

c. Set run times

d. Set water days

(This is the correct sequence). See table 4.3.

Pre-test Scores Post-test Scores0

1

2

3

4

5

6

Q1: Program Sequence

Table 4.3

During the pre-test, only two learners answered correctly. In the post-test, all learners responded

correctly.

Question 2: Match the controller feature with its corresponding description. See table 4.4.


Pre-test Scores Post-test Scores0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Q2: Controller Features

Table 4.4

During the pre-test, only one learner answered correctly. In the post-test, four learners responded

correctly.

Question 3: Benefits of an irrigation controller. See table 4.5.

1 20

1

2

3

4

5

6

Q3: Benefits of an irrigation controller


During the pre-test, only two learners answered correctly. In the post-test, all learners responded

correctly.

Question 4: Classification of irrigation controllers. See table 4.6.

1 20

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Q4: Controller Classification

During the pre-test, no learner answered correctly. In the post-test, four learners responded

correctly.

Question 5: Controller Types. See table 4.7.


1 20

1

2

3

4

5

6

Q5: Controller Types

During the pre-test, one learner answered correctly. In the post-test, all learners responded

correctly.

Usability Questionnaire

The learners’ attitudes and opinions of the training experience overall were quite positive.

Forty percent of the learners found that the instructions were very easy to follow. One learner felt

that the instructions were not clear and no one had a neutral response. See table 4.8.

Were the instructions easy to follow

Very Easy Easy Neutral Difficult Very Difficult


Table 4.8

Eighty percent of the learners felt that the training very significantly contributed to

increment their knowledge in irrigation while twenty percent felt it significantly contributed. See

table 4.9 below.

Significant contribution to knowledge in irri-gation

Very significant Significant UndecidedInsignificant Very insignificant

Table 4.9

The level of confidence that the learner felt to program a controller. Eighty percent felt

very confident, while twenty percent felt confident. See table 4.10


Provided confidence to program a controller

Very Confident Confident UndecidedUnconfident Very Unconfident

Table 4.10

The learners felt quite positive regarding their learning experience. See table 4.11. Sixty

percent felt absolutely no frustration, twenty percent felt hardly any frustration, and twenty

percent felt some frustration.

Frustration experienced

Lots of frustration Some frustration UndecidedHardly any frustration Absolutely no frustration

Table 4.11


Sixty percent of the learners felt that the quiz was easy, twenty percent felt the quiz were

more or less difficult and twenty percent felt that the quizzes were very easy. See table 4.12

below.

Quiz Difficulty Level

Very Easy Easy Neutral Difficult Very Difficult

Table 4.12

Something learners mentioned they liked about the training, was the detailed information

provided. Something they cannot easily find, like the programming sequence.

Something they did not like was the audio getting stuck at random times and navigation buttons

were not intuitive.

Section V

Appendices

Survey Questions 5.1

What is your ethnicity?

Where do you live?


What is your gender?

What is your age?

What do you do for work?

Have you ever worked with an irrigation controller? If yes, in what scenario? how often?

Pre-test and Post-test 5.2

Please answer the following questions:

1. Order the steps in the correct sequence:

a. Set the current time and date

b. Set start times

c. Set run times

d. Set water days

(This is the correct sequence)

2. Match the feature with its corresponding description

Manual-One Station

(Una Estación)Permite activar un solo riego para una sola válvula

Start Times

(Tiempos de inicio)

Permite establecer de 1 a 4 tiempos de inicio para cada

programa

Run Times

(Tiempos de Riego)

Permite establecer el tiempo de riego para cada válvula

entre 1 minuto y 4 horas

Water Days Permite seleccionar los días o intervalos de riego


(Días de Riego)

Seasonal Adjustment

(Ajuste Estacional)

Permite hacer cambios a los tiempos de riego sin

reprogramar el controlador

Sensor ByPass

(Ignorar Sensor)Desactiva el Sensor de Clima

(This is the correct match)

3. Multiple Choice. Which of the following options are benefits of having an irrigation

controller?

a. Provides a beautiful landscape

b. Reduces water waste

c. Saves money

d. Provides convenience

e. All of the above

(The correct answer is e)

4. True/False. A residential controller can be used in commercial applications and, a

commercial controller can be used in residential applications.

(The correct answer is True)

5. Match the classification with its corresponding type of controller:

Use Residential and Commercial

Technology SMS and ET

Power CA and Battery

Features Zones, connectivity, sensors


(This is the correct match)

Training Module 5.3

http://itcdland.csumb.edu/~rsolanorodriguez/IST622/index.html

Questionnaire 5.4

1. Were the instructions easy to follow? Please choose your response from the drop down

menu. Circle only one option. Very easy. Easy. Neutral. Difficult. Very difficult.

2. How significant did the training contribute to your knowledge about irrigation

controllers? Circle only one option. Very significant. Significant. Undecided.

Insignificant. Very insignificant.

3. How confident do you feel in programming an irrigation controller?

Circle only one option. Very confident. Confident. Undecided. Unconfident. Very

unconfident.

4. Did you experience any frustration while learning with the module? Circle only one

option. Lots of frustration. Some frustration. Undecided. Hardly any frustration. Absolutely

no frustration.

5. How would you describe the level of difficulty of the quizzes?

Circle only one option. Very easy. Easy. Undecided. Difficult. Very difficult.

6. What did you like about the instruction? Please type your answer.

7. What did you dislike about the instruction? Please type your answer.

Observation Checklist 5.5

http://itcdland.csumb.edu/~rsolanorodriguez/IST622/index.html


Checklist Yes No Comments

Does the learner complete every

item in the module?

4 1 One learner expressed he wasn’t

sure what to do.

Does the learner follow the

instruction correctly?

4 1 One learner was not familiar with

the drag and drop feature.

Does the learner have any

difficulty following instructions?

3 2 Some learners were not sure what

to do. This was more noticeable on

the quiz questions.

Does the learner follow the

Module navigation without

difficulty?

3 2 Some learners had difficulties

working with the navigation

buttons.

Does the learner encounter any

problem, but manages to solve it

on his/her own without developer’s

intervention?

1 4 Yes, one learner noticed the audio

getting stuck but realized that if he

clicked again, the audio had a

better chance to finish.

Does the learner encounter any

problem, and need developer’s

intervention to be able to continue?

1 4 One learner was not familiar with

computer use and required more

explanations to be able to advance.

rsinstructionaldesign.comrsinstructionaldesign.com/ist 622/solano ist622 project.docx · web...

Documents