robert riordan + sprott school of business for cu …...robert riordan + sprott school of business...

Robert Riordan + Sprott School of Businessfor CU EDC

Towards the development of a self-assessment metric for TA performance:

We have 120 minutes together today Break at around 60 minutes… The logins for the machines are same as any

lab on campus – your Carleton Connect credentials

Washrooms & exits Question format

Anytime, anywhere :-)

Am I doin' good? © Robert Riordan ~ 2009 - 2011 2

First me, then…

You

Department

Research interest, thesis topic

What your role as a TA involves

What feedback mechanism is in place for you… ?


Attend 10 workshops Write two workshop response papers (one

can be for this workshop) Write one TA Talk Article

Have you formally registered?

This applies to continuing/experienced TAs

More information at www.edc.carleton.ca


Symbols Ouch :( Background & Performance Some caveats Information warm up – Do the Fermi Incentive Schemes Direct Data Collection Methods

Questionnaires/Surveys/Interviews Anecdotal Sample size determination Question Types

Indirect Data Collection Methods Instrumentation Types of measures

Hands-on Blogger Survey Monkey

Other Technology I’ve got feedback, now what? Summary


There are a couple of symbols used throughout this presentation…

Participation – we talk

Hands-on – we do


The worst teacher ever! Such a stupid and Self-righteous Prof.!!!! If my guessing was right, I think the most of the well rating about him were made by himself. How can I just describe about him!? Ugly, self-fish, rude, Shameless, and so on. Stay far far far away from him! If you could see this, please stop rating yourself anymore!


Often, we receive only negative feedback

Get a thick skin ASAP!

How well do you take criticism?


Though we might not always want feedback

We all need feedback

We all want to do a good job

If we don’t get feedback, we make it up

Anyway, who cares…?

The answer is that we do care and the institution cares and for sure, your students care!


From an instructor perspective…

It’s not always possible to give good feedback to TAs, even if one wants to, owing to:

▪ Time constraints

▪ Communication issues

▪ The sometimes subjective nature of the work we do

▪ Personality stuff – can neither give nor receive neither praise nor criticism

▪ Asynchronicity of the structure in which we work


We will examine several strategies and tools that you can use to get an idea of how you are performing in the absence of what you consider to be satisfactory feedback from your supervisor

This stuff can be used in isolation or in combination with other feedback mechanisms to provide a more clear picture of your performance


You should always secure permission before embarking on any interaction with your constituents

Such work might be UNPAID!


Will a prof hang you out to dry? Let’s be realistic

▪ Teaching ratings are an important component of a prof’s own feedback loop

▪ They are important for tenure, for progression through the ranks and for employment opportunity

▪ Some might use you to deflect criticism from themselves

But let’s be honest, too…▪ It doesn’t happen very often at all…

▪ Don’t be hyper-aware of stuff like this to the point of poisoning your relationship


Some material has been adapted from:

How to Measure Anything: Finding the Value of Intangibles in Business, 2nd Edition

Douglas W. Hubbard

John Wiley and Sons, 2010


http://www.howtomeasureanything.com/

You tell me…


A quantitatively expressed reduction of uncertainty based on one or more observations

Douglas W. Hubbard

Anything that can be observed can be measured

No matter how fuzzy the measurement, anything that reduces uncertainty is a measurement

So-called intangibles are measurable, too

Happiness

Satisfaction

Impact


Guy named Einstein: “Not everything that can be counted counts, and not everything that counts can be counted.”

Enrico Fermi, Italian physicist

A father of the atomic bomb along with Robert Oppenheimer

Famously estimated the number of piano tuners in Chicago using a series of guestimations

Point is that the results of any honest measurement effort will yield a better estimate than saying “I have no idea.”



Either individually or in groups of 2… Use what you know to estimate: # of litres of milk (chocolate + white from skim up to

homo) consumed in Ottawa in 2010

# of cigarettes smoked by youth (15-19 years old), by gender, in Ontario in 2009

# of A grades (A-, A, A+) achieved by all Carleton students in the fall term of academic year 2010

Population hints: Ottawa: ~ 900,000 (5th most populous in Canada and 2nd in Ontario)

Ontario: ~ 13,000,000 (largest in Canada and ~5th largest in NA), Canada 33,000,000

Carleton: ~ 25,000


Milk (litres/person 2007):

2% – 38.0

1% – 18.3

3.25% – 12.0

Skim – 8.8

Chocolate – 5.7

Total ~ 83 * 900,000 = 74,520,000

Source: http://wendihiebert.com/2009/02/19/dairy-data-canadas-declining-per-capita-milk-consumption/


Cigarettes smoked by youths (2008 data) 14.8% were current smokers

▪ 8.6% were daily and 6.2% non-daily smokers

Daily smokers averaged 12.2 cigarettes/day▪ Assume non-daily smokers smoke at 1/3 of daily smokers’ rate = 4

cigarettes/day

16.8% of males and 12.7% of females were current smokers Population aged 15-19 on July 1, 2009

▪ Male: 1,153,334 ~ Female: 1,098,791

Total: ~> 1,000,000,000 (Canada); 420,000,000 (Ontario)

http://www.tobaccoreport.ca/ytu.htmlhttp://www.statcan.gc.ca/daily-quotidien/091127/t091127b2-eng.htmhttp://www.statcan.gc.ca/daily-quotidien/090326/t090326a2-eng.htm


http://www.tobaccoreport.ca/ytu.html

http://www.tobaccoreport.ca/ytu.html

http://www.statcan.gc.ca/daily-quotidien/091127/t091127b2-eng.htm






http://www.statcan.gc.ca/daily-quotidien/090326/t090326a2-eng.htm






Grades…


0

5

10

15

20

25

F D C B A

Once you have permission, determine if: Prof will allow bonus marks for students to

participate in feedback What are the issues with incentive? Make sure you have a mechanism to track unique

student evals – duplicates are bad data

Self-selection bias (non-random sample):▪ Students who participate will likely fall into two groups:

▪ High achievers (need good grades for their own feedback loop )

▪ Low achievers (desperate to pass in any way possible)

▪ The great mass in the middle is missing


These measurements require direct contact with the student in order to gather data

They require explicit permission from the instructor or department Make sure there are no privacy violations

Strongly recommend that you get approval of the instrument before administering it

Direct contact can be either anonymous or identified There are issues with each…


Email How to harvest emails?

Web Course website

WebCT

Your own site

Blog Online surveys

PHYSICAL DISTRIBUTION

In class or in lab or tutorial session or office hours?

Inserted in tests/exams? Included in notes/lab

material? Others?

ELECTRONIC DISTRIBUTION


PHYSICAL COLLECTION

Not anonymous but almost… especially in large class settings or with drop boxes Drop box at department

office?▪ With permission

Drop box at your office?▪ Security?

Submit along with test/assignment/lab?

Others?

ELECTRONIC COLLECTION

If not part of the tool (as in an online survey)

Can be anonymous Returned as body of or as

attachment to email

Anonymous issues Can’t control:

Multiple submissions No submission Quality of responses


Create an instrument from which it is easy to extract data:

Numeric/multiple choice/Likert scale questions…

Fewer items the better

Decide on a focal element or two and stick to them!

Example foci:

▪ Turnaround speed

▪ Richness of feedback

▪ Availability

▪ Others…


When to interview Office hours

Test/assignment/lab reviews

Before or after class or during a break

Scheduled time▪ By invitation (you invite at specific time) or by appointment

(their initiative)

Environment is important▪ Confidentiality is important

▪ Comfort is important

▪ Beware the interviewer effect – do you control their marks?


Simply ask students about their experience: Satisfaction with your service

▪ Be sure to separate you from the rest of the infrastructure

▪ This will be important no matter which direct measure you use▪ Students sometimes don’t differentiate you from the prof from the lectures

from the lab machines from the textbook…

Difficult to get an empirical measure from this technique▪ Can’t compare across interviewers as it’s very free-form.

Can suggest topics for a more focussed instrument (in the Bayesian sense)

Again, be aware that people often use an opportunity like this to vent… you are an easy and available target


Sample size

This can be simple or sophisticated… let’s be simple about it:

▪ Let’s assume that you have an item such as the following: I am satisfied with the service provided by the TAs in this class. Agree

or Disagree?

▪ What does it mean if 62% agreed? Or if 29% disagreed?

▪ How many people do I need to ask?

▪ What’s the margin of error?


Let’s start at the obvious extremes in terms of sample size If your sample was ONE and that person disagreed, you’d

have a dismal satisfaction rate of 0% OR a dissatisfaction rate of 100%. Time to find the O-Train trestle?

Obviously not…

Facts: Required sample size is a function of three simple things: Confidence level Confidence interval Population size


Confidence level

Do you want to be 95% sure? Or 99% sure?

Sure of what?

Confidence interval

Sure that your observation is +/- a CI of the true population figure (the real, underlying answer to the question that would be exposed if you interviewed everyone in the population of interest)

Population size

From how many are we sampling?


Example with lexicon: If 68% agreed with your statement and you want a:

▪ Confidence level of 95% and▪ a confidence interval of 5▪ from a population of 200▪ You would need 132 observations (persons asked) to be able to

state that:I am 95% sure that between 63% and 73% of students in this class agree

that they are getting good service…

Stated another way:▪ If you did 100 surveys of 132 students from this population of 200,

95% of the time the observed percentage agreeing would be between 63 and 73. Five times it would be outside this window, even if the true population percentage is 68%.


Let’s go to a website and fiddle around… http://www.surveysystem.com/sscalc.htm

Level Interval Population Sample Size?

95 5 60 52

95 5 300 169

95 1 60 60

99 5 300 207

95 5 30,000,000 384

99 1 30,000,000 16,632


http://www.surveysystem.com/sscalc.htm

Sometimes we don’t have the luxury of being able to specify sample size

So we can work backwards Given the numbers we have, how do we

estimate confidence in the measurement?


This table shows the 90% CI for small samples (Lower Bound and Upper Bound shown as %) Simply look up the sample size in the column and the row with the number of hits. To estimate more than 10 hits, invert the table and estimate ‘misses’ …

1 2 3 4 6 8 10 15 20 30

0 2.5-78 1.7-63 1.3-53 01.0-45 0.7-35 0.6-28.3 0.5-23.9 0.3-17.1 0.2-13.3 0.2-9.2

1 22.4-97.5 13.5-87 9.8-75.2 07.6-65.8 05.3-52.1 4.1-42.9 3.3-36.5 2.3-26.4 1.7-20.7 1.2-14.4

2 36.8-98.3 25-90.3 18.9-81 12.9-65.9 9.8-55 07.9-47.0 5.3-34.4 4.0-27.1 2.7-18.9

3 47-98.7 34.3-92.4 22.5-78 16.9-66 13.5-57 9.0-42 6.8-33 4.5-23

4 55-99.0 34.1-87 25.1-75 20-65 13-48 9.9-38 6.6-27

5 48-94.7 34.5-83 27-73 17.8-55 13.2-44 8.8-31

6 65-99.3 45-90 35-80 22.7-61 16.8-49 11.1-35

7 57-95.9 44-87 28-67 21-54 14-38

8 72-99.5 53-92 33-72 25-58 16-42

9 64-96.7 39-77 29-63 19-45

10 76-99.6 45-82 33-67 21-49

Sample Size

Nu

mb

er o

f “h

its”

in S

amp

le

...



Two basic genres of question: Quantitative (hard metric)

▪ What was your age at your last birthday?

Qualitative (open-ended)▪ How do you feel about getting older?

There are pros and cons to each… Qualitative is more time consuming to analyse

and is difficult to summarise

Quantitative is easier to work with but not nearly as rich… nothing is free.


When you can’t get direct access to students either personally or anonymously…

Indirect measures are sometimes called proxyor surrogate measures because they stand infor direct measures…

Will need approval (and active participation from your department as well) to get some of these data…

Some can be had for the price of a coffee :-)


Compare anecdotally with other TAs Across sections of the same course in the same term Across sections of the same course in different terms

(if possible – see below for a solution going forward) Across different courses with similar structures in the

same or different terms Start a process/data repository to keep

measurements for future reference Great longitudinal data source – can be used for

several purposes (benchmarking, marketing, etc…) Maybe your department would be interested in

supporting such an enterprise?


Instrumentation

Either alone or in a group, you can develop measures to give proxy indicators of your efficiency and effectiveness

Speed:

▪ Minutes per task (per test, per paper, per experiment)

Accuracy:

▪ Number of returns, complaints, appeals, etc

If you use speed, you really must also use accuracy

Speed is a relative term… faster is not always better!


Instrumentation (cont) If you are comparing across sections or years or

instructors, you really must control for (take into account) certain potential and implied differences…

These include:▪ Instructors▪ Curriculum▪ Time of day (huge effect on class demographics and

satisfaction!)▪ Term (summer is way different!)▪ TAs▪ Environment (room, lab, etc.)▪ Others?


What kind of measures can you create? Attendance rates for lectures, labs, tutorials,

office hours▪ Need total numbers for denominator

▪ Can average them over a term to remove peaks caused by approach of tests/exams/assignments

▪ Can remove the peak days and look at ambient levels

▪ You are looking for patterns and trends here

▪ This type of data is difficult to interpret in isolation and must be considered in terms of the contaminants mentioned on the previous slide


Measures (cont) Rates of contact by type of contact:

▪ Argue about mark

▪ Request for remark

▪ Clarification

Various denominators are possible, viz:▪ Total students in class, lab, etc

▪ Total students for whom you are responsible

▪ Total visits/contacts

▪ Others?


Measurements (cont)

Average and maximum discrepancy in grade adjustment, by various denominators, viz:

▪ Total students

▪ Total adjustments made

▪ Others?


Secondary or administrative data With the permission of your department, you

can use aggregate data such as: DFW (Drop, Fail, Withdraw) rates

Grade distributions over time (this is very sensitive data…) in as fine a detail as you can get them, viz:▪ By term, instructor, enrolment, etc.

▪ By numeric (if available) and/or letter grade

▪ By time of day

▪ Any other detail available


Blogging A blog is a web log … a recording of random and/or

purposeful messages to the world hosted on a blogging site on the web

Blogs have interesting features for us▪ They are free

▪ They allow anonymous feedback

▪ They allow very simple surveys

▪ They can provide links to documents (if you have a place to host the docs)

▪ They can provide links to other, more sophisticated (but still free) online survey tools


Let’s go and take a BLOG survey now… Fire up your machine and navigate to: http://carletonta.blogspot.com

Don’t click thesurvey link yet!


http://carletonta.blogspot.com/

Now let’s create a BLOG at:

http://www.blogger.com


http://www.blogger.com/

If you have a Google (Gmail) account, log in If not, click on

Blogger requires a Google account

If you don’t have one, take a moment to create one if you want one…

From here, just follow the directions to create a blog…

I’m here for moral support :-)


Now let’s go back to the BLOGGER survey at: http://carletonta.blogspot.com

NOW click thesurvey link!


http://carletonta.blogspot.com/

Now that we have some data in there, let’s take a quick look at the output

Login

My Surveys

Analyze


Certain functions are not available on the free basic account


IF we have time, we’ll create our own survey on http://surveymonkey.com …

We can mix and match questions and question types…

Experiment with scales


http://surveymonkey.com/


Source: http://www.tagcrowd.com/

WebCT does not seem to have any tools that we can use for our purposes, other than being a vessel for providing the download of a physical survey or a link to a blog/survey…

Twitter…

Could really get out there with this technology…

Would need to designate some Tweeters in the class (perhaps incentive-based) and follow them?

Any other suggestions on how to use it?


Facebook?

Create a group and log feedback

YouTube?

Broadcast announcements…

▪ How to get feedback?


How to interpret?

That depends…

Teaching Portfolios

Academic appointments frequently require a teaching portfolio

▪ Evidence of track record and excellence in teaching including concrete EVALUATION DATA

▪ This is a great start…


Sometimes and for a variety of reasons, we don’t get the feedback that we need

If, however, we are creative, inventive, diligent and motivated, we can find and do all kinds of things in order to satisfy our innate need for feedback…

There are basically two types of data:

Direct and indirect

And responses can be anonymous or not…


You can cooperate with others, or go it alone You can use paper or electronic means You can use secondary (administrative) data

if you cannot get direct (primary) data You can start a programme of data collection

and archive in order to support the initiative of improving feedback for TAs and thus positively impacting the quality of education at Carleton… a laudable goal indeed…


It’s been a great pleasure :) My contact information:

Rob RiordanSprott School of [email protected]

Don’t be strangers! If you’d like a copy of the presentation, email

me :-)


mailto:[email protected]

robert riordan + sprott school of business for cu …...robert riordan + sprott school of business...

Documents