fellow information - rouskas.csc.ncsu.edu
TRANSCRIPT
1
Computing Across Curricula (CAC) The Final Report of Fellow Action Research Project
Fellow Information Name: Reha Uzsoy Position (Job Title): Professor
Department: Industrial & Systems Eng’g CAC Fellowship Year and Semester:
Email: [email protected] Phone: 3-1681
Years Employed at NCSU: 3 Years of Teaching (Total): 25
Courses you teach (on average) Number of Fall Sections: Undergraduate Level ___0____ Graduate Level ____1__ Number of Spring Sections: Undergraduate Level ___1____ Graduate Level ___1___
Project Information Action Research Project Title: Modelling for Computer Problem Solving
Course Information Course Title: Computer-based Modelling for Engineers
Course Number: ISE110 Number of Students: 39 at start of semester
PURPOSE & OBJECTIVES
The primary objectives of the class are to provide students with a comprehensive basis in programming skills which subsequent classes in the curriculum can build on, using the medium of Excel and Visual basic for Applications. Based on my experience sitting in the class this semester, the class does an admirable job of enhancing studentsʼ computing and problem skills, as ongoing assessment has demonstrated. However, one of the most useful skills our students acquire throughout their education is in the area of modeling - isolating the critical and noncritical components of a problem, abstracting these into a representation that is amenable to mathematical and/or computational analysis, and extracting and presenting insights from the results of the analysis. Previous studies in this class have examined the effect of the computer technology on the studentsʼ problem solving ability. However, there was no explicit component of the course aimed at enhancing problem solving ability – the benefits were all incidental. The purpose of this project is to examine whether a specific component aimed at enhancing studentsʼ problem solving skills would show benefits over the course of a semester.
METHOD Students were organized into groups of three, which will also be their teams for the class projects. In the first phase, to establish a baseline, the teams were given an appropriately
2
selected, ill-structured realistic problem description and asked to prepare a written report discussing how they could use computation to study this situation. One class session was devoted to an interactive discussion of the various proposals, for the purpose of idea sharing and illustrating the different approaches that could be taken. After this discussion, the students were asked to develop an improved model and validate it to some degree. In both these assignments, students were provided with a list of guiding questions they needed to address. The questions were aimed at making students think about several different aspects of the problem, such as the presence of multiple stakeholders with different goals, different types of uncertainty present in the problem, missing data and related issues. All groups submitted written reports, which were returned with extensive feedback, and made oral presentations to the whole class, thus allowing every group to see what the other groups had done. The studentsʼ level of problem solving proficiency was evaluated using the written reports based on Wolcottʼs Steps to Better Thinking rubric, which had also been used for the previous problem-solving studies in this class. The difference between the baseline and the second project was assessed to examine the benefit of the modeling exercise for the studentʼs understanding of basic modeling issues. The project was implemented in both the Fall 2009 and Spring 2010 semesters. Harvard Business School case studies were used to provide the problem environment in both cases. In the Fall 2009 semester, Case No. HBS 9-681-061, “University Health Services: Walk In Clinic” was used. This case study focuses on attempts to reduce the average waiting time for patients in a university health clinic while maintaining the support of different stakeholder groups including doctors, nurses and the clinic administration. In the Spring 2010 semester, in an attempt to link the project more closely to the Excel content of the class, Case 9-698-053 Hamptonshire Express was used. This latter case is essentially a case in supply chain coordination delivered through a simple two-stage system and a simple newsboy inventory model. This second case proved to be very structured in nature, and together with the extensive Excel spreadsheets provided, resulted in the project being highly structured, with little of the ambiguity and problem definition requirement that would require student problem solving skills. Hence the analysis for this study focused on the projects from Fall 2009. Copies of the assignment sheets and the grading rubrics used for the phases are included in the appendix. The study was conducted in a pre-post design. In the initial phase, the written reports submitted for the first phase were analyzed according to the Wolcott scale. This was done by the instructor, after some instruction from Dr. Raubenheimer in how to use the Wolcott rubric. The written reports for the second phase were then reviewed, after extensive feedback had been given. The basic research question was whether or not the students exhibited improved scores in the second phase of the project. Students worked on the project in teams of three. These teams were partially self-selected; some students formed groups, while others asked the instructor to assign them partners. Several teams began with three students but did the second phase with only two, due to students dropping the class over the semester. The students were mostly sophomores, but with several juniors, seniors and even a graduate student included, and highly diverse backgrounds in terms of prior computing and problem solving experience. It is also notable that the project is a relatively small portion of the course grade (8%), and the class has a very heavy workload in terms of homework, with many students also taking heavy course loads outside this class. Hence the degree of emphasis and effort students devoted to this class varied quite considerably. In some groups there
3
was real evidence of teamwork and collaboration; in many teams, one student took the lead and others contributed minimally; while in yet others the group turned in a very minimal effort. The results in the following section should thus be read with this background in mind. RESULTS In the Fall 2009 semester, 9 groups completed Phase 1, and 8 completed Phase 2. The results of the first phase of the project according to the Wolcott scale are summarized in Figure 1 below. The Wolcott rubric assigns scores on a scale of 1-5 for each of the categories indicated on the horizontal axis. As can be seen, in Phase 1student teams performed quite poorly. In the instructorʼs estimation, no team did well in terms of eliciting the relevant information; a very rudimentary understanding of uncertainty was displayed, with most groups failing to recognize the presence of uncertainty and attributing this to lack of data. One group did somewhat better than the others in terms of organizing information and laying out solution alternatives, but this success did not go beyond 2 on a five point scale. These results are to be expected in a lower level class, where students have had little exposure to problem solving except in highly structured environments.
Figure 1: Wolcott Scores for Project Teams in Phase 1 The scores obtained by the teams completing Phase 2 are shown in Figure 2 below. The results are considerably improved, although statistical significance cannot be claimed and was not explored. The average score of each team over all categories has more than doubled in several cases. The elicitation of relevant information is now scoring higher, and there is better performance across the board in interpreting and organizing information and systematically judging options. Two groups have actually addressed more advanced issues of identifying the limitations of their analysis, which was not attempted in the Phase 1 submissions.
4
Figure 2: Wolcott Scores for Project Teams in Phase 2 Figure 3 compares the average scores over all teams for each category in Phases 1 and 2. These indicate that there were apparent improvements from Phase 1 to Phase 2. This clearly cannot be attributed to more effective problem solving skills on the part of the students. The questions the students were required to answer in Phase 2 were designed to guide them to address certain of the Wolcott categories, and there are many sources of variability between groups. The absence of a control group is also a major limitation, which will need to be addressed in this work in the following semesters.
Figure 3: Comparison of Average Wolcott Scores by Category
5
CONCLUSIONS Given the critical foundation of computing skills ISE110 provides for the rest of the ISE curriculum, it is not feasible to dedicate a large amount of time to discussion of modelling, nor is it realistic to expect students to become expert modelers after brief exposure in one class. However, exposing the students to the logic of modeling has a number of advantages. Much of the later ISE curriculum involves the development and application of various kinds of mathematical models, ranging from statistical tools like regression and control charts to numerical optimization techniques like linear programming and purely computational approaches like Monte Carlo simulation. Secondly, much ISE work in industry is conducted in environments with heavy business and personnel aspects, leading to the need to build models that abstract key features of a situation that are of importance in the business context, involving non-technical aspects of the environment that must be considered for the solution to be successful. Finally, ISE 110 provides an environment where, once basic computing skills have been acquired, students can implement the models they develop and perform some basic validation of the models against data obtained from the real system, or from some surrogate for the real system provided by the instructor. The results of this project are clearly limited in scope, but suggest that students can at least be made aware of the need for a structured approach to problem solving, which can help them do better than when they are left to flounder without guidance. The experience of being faced with a seemingly chaotic situation and finding that they can make sense of it and suggest meaningful solutions is a strong motivator and confidence builder for some students, while others are content to go through the motions and show little interest. In this aspect, these students were no different from the senior and masters student the instructor has applied this project approach to in prior years. A few broad observations are that students concept of uncertainty is very limited, which is to expected to some degree since many have not been exposed to probability and statistics; it was interesting that several students who were taking probability did not appear to relate it to the project until forcibly made aware of the relation. The ability to view the problem from multiple perspectives was also something lacking in most groups, although the project guided them through this experience it is not clear how well this was internalized.
NEXT STEP/LESSONS LEARNED The results of this preliminary study suggest that the exposure of low level students to
problem solving issues can yield benefits, if only in making them aware of the issues they need to consider. As suggested by several texts on teaching problem solving, providing students with a systematic procedure with which to approach ill-structured problems has benefits. While the nature of ISE110 is such that a large portion of the class cannot be devoted to this issue, it appears perfectly feasible to include such exercises as a minor portion of the class, exposing students to issues that can be further developed throughout the rest of the undergraduate curriculum. For this approach to yield maximum benefit, however, subsequent courses must maintain these themes and build upon these concepts.
6
In the immediate future, we plan to continue refining the presentation and design of these projects to support problem solving skills. In Fall 2010 the course instructor, Ms. Kuang-Hao Yeh, is planning an alternative design where students will be presented with a problem, and must submit a formal proposal describing how they will use the computing technology taught in the class to develop a solution. They will then work in teams to develop the solution and evaluate its effectiveness. Another interesting direction will be to use the Wolcott approach to assess the degree of problem-solving competency in a masters level class on supply chain management, where three such projects constitute 25% of the grade. Finally, the assessment of the impact of these projects on student problem solving ability needs to be conducted with a formal control group, to help achieve a clearer understanding of to what degree the results are due to the projects, and what to other environmental factors.
7
APPENDICES
ISE 110 Computer Modeling for Engineers
Fall 2009 – Dr. Reha Uzsoy
Term Project Part I
Due Tuesday, September 29, 2009
Overview: The purpose of the class project is to give you hands‐on experience with
taking a realistic, practical engineering problem situation, determining what the
problem is and what constitutes an acceptable solution, analyzing it quantitatively
using the computer tools presented in class, and recommending solutions that are
implementable. This involves a number of different areas: different people involved
will have different ideas as to what the problem is, and what the solution ought to
be; all the data required may not be available, especially data regarding the future;
and there may exist a range of possible solutions that may all be reasonable. This is
precisely the type of problem‐solving activity that will be required of you in your
professional practice. Also, consistent with professional practice, you will be
required to work in a team, and present your results both orally and in a formal
technical report.
All questions, discussions and grading regarding the project will be
handled by Professor Uzsoy; apart from technical issues of using Excel, the TA
will not be involved in the project.
The Problem:
The costs of healthcare are currently the focus of an intense national debate,
so we shall consider a problem in healthcare management: how to organize the
operations of a university walk‐in clinic, described in the Harvard Business School
Case Study No. 9‐681‐061, “University Health Services: Walk‐In Clinic”. The case will
be available from the campus bookstore for your purchase; it is part of the required
material for the class.
Project Team:
You will work on this project in a team of three students. Please submit the
name of one person you would prefer to be teamed with to Professor Uzsoy by
Tuesday, September 15, 2009. We will then form teams of three people, trying our
best to respect everyone’s preferences. However, bear in mind that it is impossible
to satisfy everyone! Once teams are formed, they are fixed for the remainder of
the semester!
Our Approach:
We will work on the project in two separate phases. The first phase focuses
on exploring the problem with your team, reaching a consensus among yourselves
as to what the problem is, what a good solution would look like, and performing
some basic data analysis (yes, with Excel!) to develop an idea of what solutions may
be possible. Each team will make a formal Powerpoint‐supported presentation in
class summarizing the results of this analysis on Tuesday, September 29, 2009. This
will be accompanied by a written report on your analysis and conclusions submitted
the same day. I will work with all teams in helping to develop the presentation and
report before its submission. I will also meet with each team after the presentation
to discuss the presentation and report, answer questions, and suggest
improvements.
In the second phase of the project you will build on your initial analysis to
develop a set of recommendations to clinic management on how to improve the
operations of the clinic, and justify these based on cost and performance. This will
require addressing the concerns of the different constituencies involved, as well as
practical issues to make your recommendations implementable in practice. This
second phase will involve an in‐class discussion on Thursday November 5, 2009; a
second presentation on Tuesday December 1, 2009; and the final written report
on Thursday December 4, 2009. Again, I will work with your team as you develop
both report and presentations.
Project Phase 1:
This phase of the project is aimed at becoming familiar with the problem,
identifying what is needed and what makes it difficult to solve, and doing some
initial, exploratory data analysis to set the stage for your more complete analysis in
the later phases.
Exploring the Problem:
1) Who are the different stakeholders in the problem who may have different
ideas as to what a good solution is? Obviously, the clinic management, as
represented by Angell, is one such; what other groups with different
perspectives need to be considered? In other words, who are we trying to
make happy, or at least, less unhappy?
2) For each group of stakeholders you identified in Question 1, describe their
idea of a “good solution”. Remember people will have different incentives,
such as financial rewards, job satisfaction, etc. Hence for a solution to be
implementable in practice, i.e., all the main stakeholders need to be willing to
at least “live with it”.
3) List the data items given in the case that you feel are relevant to obtaining a
good solution, together with one or two sentences explaining why each item
is needed.
4) Is there any data that is not given but you wish you had? List these items, and
explain how they would help your analysis if they were available.
5) List the primary sources of uncertainty in the problem. One way to think of
this is to think of items that if you knew them perfectly now, your solution
would always operate exactly as planned.
6) Based on your responses to these questions, summarize in at most two
sentences what it is that makes this problem difficult – why is Angell
struggling so much?
Exploratory Analysis:
7) The clinic has just moved from a pre‐triage system to implementing a triage
system. Has the triage system yielded the expected improvements? What
stakeholders are unhappy about it, and why? How would you use the data in
the case to examine what might be causing the dissatisfaction?
8) How does the demand for MD and NP services vary within a day, and across
days of the week? Estimate the average utilization of the MDs for each day of
the week, and for each morning and afternoon period. Do the same for the
NPs. Is there sufficient NP and MD time available to serve the demand
throughout the week? Relate this analysis to the waiting times incurred by
patients. Does the clinic really have the MD/NP capacity that it is supposed to
have (see p.2 of case, first full paragraph)(Hint: This is an EXCEL question;
use worksheets to do the analysis, and graphs/charts to present the results.)
9) What do you think are the problems caused by the walk‐in appointments?
Justify your answer clearly and concisely based on the information from the
case. Discuss this issue from the perspectives of the different stakeholders
affected.
Report Format:
Your report should be submitted as a Word or PDF file, appropriately spell‐
checked, formatted and prepared in a professional manner. It should consist of no
more than six double spaced pages in 12 point font, organized by the question
numbers above. You may include charts, tables and graphs in the body of the report
as you see fit, or in an appendix which must be clearly labelled and cross‐referenced
with the main body. As a rough guideline, try initially for half a page for each of the
questions above, which will leave you some room for graphs etc. There is no page
limit on the Appendix, but be warned – I will read the main body, and use the
appendix mainly for clarification of details. Both the report and the presentation will
be due in class Tuesday September 29, 2009.
ISE 110 Computer Modeling for Engineers
Fall 2009 – Dr. Reha Uzsoy
Term Project Part II
Due Tuesday, December 1, 2009
Project Phase 2:
Your first task in the second phase of the project is to revisit your responses
to the questions raised in Phase I, responding to the concerns raised in the
presentations and in the comments and discussion of your written responses. This
material should now be converted into the introduction of a technical report, with
the following sections:
Introduction: A high-‐level overview of the context for the project; why is it
being undertaken, what was achieved, and concluding with a brief overview of the
remaining sections of the report. (1.5-‐2 pages)
Problem Environment: Who are the stakeholders, what are the perceived
problems by each group, what steps have been taken to address these recently
(triage!), and data-‐driven discussion of whether they have been successful. (2-‐3
pages)
Analysis: Using the data given in the exhibits, does the clinic have enough
capacity to process the number of patients arriving within desired wait times in a
consistent manner throughout the week? (4-‐5 pages)
Recommendations: What steps should clinic management take to address
the problems they face, both in the short term and the longer term, to improve
things? You need to justify these recommendations base don the data in the case and
the analysis you have performed.
A sample report will be posted on the Moodle site for you review.
Analysis:
The analysis in this phase of the project will proceed in two stages. In the
first, you will extend the simple, average-‐based analysis you have done in Phase 1 to
examine whether the clinic has sufficient resources to treat the arriving patients. In
the second stage, you will enhance the analysis using simple queuing concepts given
below. Finally, you will explore a number of specific questions using both analysis
approaches and make additional recommendations based on your analysis.
Basic queuing concepts:
The basic idea of queuing models is simple: customers arrive at a server
according to some random pattern over time, and the server serves them, also
taking some random time to perform each service. The easiest way to envision this
is to think of a bank where a teller must serve customers arriving over time. We
don’t know exactly when customers arrive, nor do we know beforehand exactly how
long each customer will take to complete their transaction. Queuing models proceed
by assuming that the time between customer arrivals (interarrival time) and the
time to process a customer (service time) follow some statistical distribution, and
try to draw conclusions about some critical performance measures.
While there are many different queuing models that have been studied over
the last fifty years, many for specialized applications like telecommunications
networks (many of the protocols that run the Internet were developed by
researchers in queuing, such as Leonard Kleinrock of the University of California),
we shall focus on one such model that is adequate to our purposes. To use this
model, we will assume we know the mean and standard deviation of two quantities:
the interarrival time (mean ta, standard deviation sa), and the service time (mean te,
standard deviation se). We shall define one intermediate quantity – the coefficient of
variation, defined by the ratio of the standard deviation to the mean. Thus, the
coefficient of variation of the interarrival times is given by ca = sa/ta; that for the
service times by ce = se/te.
A key concept in queuing analyses is that of the average utilization, which is
given by u = te/ta; the ratio of the average service time to the average interarrival
time. In order for a queueing system to be well-‐behaved, we must have u < 1 (What
happens if u >1 on average over a long period of time?) The utilization corresponds
to the fraction of time a resource is busy over a long period of time; most queuing
analyses assume the system will be running in its current configuration for a long
time, and try to estimate average performance over this time frame.
Under these assumptions we can estimate the key quantity of interest for
Angell which is the average time a patient may be expected to wait, as follows:
T = (ca2 + ce
2 )2
( u1− u
)te + te .
Another important relationship in queuing, which holds under very general
conditions, is Little’s Law, which states that the average Number of customers in
queue = T/ta.
We shall now use these concepts to examine the problems faced by the Walk-‐
In Clinic. For more information on queuing models, a good reference is (Hopp and
Spearman 2001).
Guidelines for Analysis
Your report and presentation should make specific recommendations to
Angell as to what she can do to improve the operations of the clinic to satisfy the
various stakeholders. Your conclusion s and recommendations need to based as far
as possible on the data in the case, and on the analyses you perform using this data.
You are expected to support your arguments with tables and graphs as appropriate;
in Excel, you have a powerful tool for performing the analysis and displaying the
results. I will be available for consultation on an ongoing basis, both in office hours
and by email, for your questions regarding the project, from now until the due date.
Specifically, I will be glad to
Your report needs to address at least the following issues:
1) Revisit your analysis from Phase 1 to estimate the average utilization of MDs
and nurse practitioners in each time period each day of the week. Display
these results in an appropriate table (Conditional formatting may be
helpful…).
2) Based on this analysis, identify some short-‐term steps Angell can take to
improve things.
3) It is clear from your analysis in Phase 1 that the clinic has an issue with
limited MD capacity. Recall that the MDs are supposed to be working 12
hours per week in the clinic. Examine the data in the case to determine
whether this is, in fact, the case. If this is not the case, how would the
situation change if they were able to get all the MD capacity they are
supposed to have? Use your calculations from 1) above to address this.
4) Develop a rough-‐cut queuing analysis to estimate the average wait times for
an MD using the relationships given above. Assume initially that ca = ce = 1;
this corresponds to an exponential distribution of service and inter arrival
times, which is generally a reasonable, somewhat conservative assumption in
service systems of this nature. Set these calculations up in Excel; you’ll need
to reuse them for other portions of your analysis. Be sure to explain how you
calculate the various parameter values clearly, and state explicitly all
assumptions you make.
a. To guide you, consider the following steps:
b. Estimate the average arrival rate of patients (patients/hr) to the clinic
for the entire year.
c. Estimate the average service time over all MDs; remember to factor in
the fact that MDs are not available all week, but only limited hours.
d. Estimate the average time in queue from the equation above. (What is
the relation between average time in system and average time in
queue?)
e. Compare the rate at which different MDs serve patients, and use this
data to estimate the standard deviation of the service time seen by a
random patient walking in at a random time to see a random doctor.
Recalculate your estimate of the average waiting time using this
information, and discuss its implications for Angell.
f. What are some realistic options that Angell and the clinic
management can pursue to reduce the workload on the MDs? Discuss
the possible effects of such decisions on the average waiting times
using the queuing model you have developed above.
g. What are the implications of these average waiting times for the
number of patients waiting at the clinic at a given time? (Think space
in waiting rooms…)
Report Format:
Your report should be submitted as a Word or PDF file, appropriately spell-‐
checked, formatted and prepared in a professional manner. It should consist of no
more than 10 double spaced pages in 12 point font, organized by the sections above.
You may include charts, tables and graphs in the body of the report as you see fit, or
in an appendix which must be clearly labelled and cross-‐referenced with the main
body. As a rough guideline, try initially for half a page for each of the questions
above, which will leave you some room for graphs etc. There is no page limit on the
Appendix, but be warned – I will read the main body, and use the appendix mainly
for clarification of details. The presentation will be due in class Tuesday
December 1, 2009; the report Thursday, December 3, 2009.
References
Hopp, W. J. and M. L. Spearman (2001). Factory Physics : Foundations of Manufacturing Management. Boston, Irwin/McGraw-‐Hill.
Oral Presentation - 40 Points
Organization of topics (10):
Presentation of information (10):
Observation of time limits (10):
Preparation of presenters(10):
ISE 110 Computer-Based Modeling for EngineersFall 2009 - Dr. Reha UzsoyProject Phase 2 Reports
ISE110 Computer-Based Modeling for EngineersFALL 2009 - Dr. Reha Uzsoy
Project Phase II Written report
GROUP:
Introduction: out of 10 points:! Overview of company background, summarizing the business need for the ! project:! Brief overview of project relative to business needs: ! Outline of rest of report:.
Problem Environment: out of 10 Points! Stakeholders:! Perceived Problems by groups: ! Recent efforts to address problems:! Data driven assessment of success:
Analysis: out of 45 Points! Describes symptoms of problem;! Suggests hypotheses as to causes.! Appropriate analysis to test hypothesis, with appropriate use of available data: ! Assumptions clearly explained and justified. ! Appropriately organized - different aspects of analysis presented in a logical, ! integrated manner.
Conclusions and Future Recommendations: out of 15 Points! Effective, concise summary of the results of the analysis - what were the ! problems?.! Recommended solutions:! Suggestions for next steps after current study: N/A
Presentation and Writing: out of 20 points! Grammar, spelling:! Sentences, paragraphs; ! Organization and transitions between sections:!Integration of figures and tables into text:! Use of appendices. N/A
Total Points:
Steps for Better Thinking Rubric ←Less Complex Performance Patterns More Complex Performance Patterns→
Steps for Better Thinking SKILLS
"Confused Fact Finder" Performance Pattern 0—How performance might appear when Step 1, 2, 3, and 4 skills are weak
"Biased Jumper" Performance Pattern 1—-How performance might appear when Step 1 skills are adequate, but Step 2, 3, and 4 skills are weak
"Perpetual Analyzer" Performance Pattern 2—-How performance might appear when Step 1 and 2 skills are adequate, but Step 3 and 4 skills are weak
"Pragmatic Performer" Performance Pattern 3—-How performance might appear when Step 1, 2, and 3 skills are adequate, but Step 4 skills are weak
"Strategic Re-Visioner" Performance Pattern 4—-How performance might appear when one has strong Step 1, 2, 3, and 4 skills
Step 1: IDENTIFY A—Identify and use relevant information B—Articulate uncertainties
A0—Uses very limited information; primarily "facts," definitions, or expert opinions
B0—Either denies uncertainty OR attributes uncertainty to temporary lack of information or to own lack of knowledge
A1—Uses limited information, primarily evidence and information supporting own conclusion*
B1—Identifies at least one reason for significant and enduring uncertainty*
A2—Uses a range of carefully evaluated, relevant information
B2—Articulates complexities related to uncertainties and the relationships among different sources of uncertainty
A3—Uses a range of carefully evaluated, relevant information, including alternative criteria for judging among solutions
B3—Exhibits complex awareness of relative importance of different sources of uncertainties
A4—Same as A3 PLUS includes viable strategies for GENERATING new information to address limitations
B4—Exhibits complex awareness of ways to minimize uncertainties in coherent, on-going process of inquiry
Step 2: EXPLORE C—Integrate multiple perspectives and clarify assumptions D—Qualitatively interpret information and create a meaningful organization
C0—Portrays perspectives and information dichotomously, e.g., right/wrong, good/bad, smart/stupid
D0—Does not acknowledge interpretation of information; uses contradictory or illogical arguments; lacks organization
C1—Acknowledges more than one potential solution, approach, or viewpoint; does not acknowledge own assumptions or biases
D1—Interprets information superficially as either supporting or not supporting a point of view; ignores relevant information that disagrees with own position; fails to sufficiently break down the problem
C2—Interprets information from multiple viewpoints; identifies and evaluates assumptions; attempts to control own biases*
D2—Objectively analyzes quality of information; Organizes information and concepts into viable framework for exploring realistic complexities of the problem*
C3—Evaluates information using general principles that allow comparisons across viewpoints; adequately justifies assumptions
D3—Focuses analyses on the most important information based on reasonable assumptions about relative importance; organizes information using criteria that apply across different viewpoints and allow for qualitative comparisons
C4—Same as C3 PLUS argues convincingly using a complex, coherent discussion of own perspective, including strengths and limitations
D4—Same as D3 PLUS systematically reinterprets evidence as new information is generated over time OR describes process that could be used to systematically reinterpret evidence
Step 3: PRIORITIZE E—Use guidelines or principles to judge objectively across the various options F—Implement and communicate conclusions for the setting and audience
E0—Fails to reason logically from evidence to conclusions; relies primary on unexamined prior beliefs, clichés, or an expert opinion
F0—Creates illogical implementation plan; uses poor or inconsistent communication; does not appear to recognize existence of an audience
E1—Provides little evaluation of alternatives; offers partially reasoned conclusions; uses superficially understood evidence and information in support of beliefs
F1—Fails to adequately address alternative viewpoints in implementation plans and communications; provides insufficient information or motivation for audience to adequately understand alternatives and complexity
E2—Uses evidence to reason logically within a given perspective, but unable to establish criteria that apply across alternatives to reach a well-founded conclusion OR unable to reach a conclusion in light of reasonable alternatives and/or uncertainties
F2—Establishes overly complicated Implementation plans OR delays implementation process in search of additional information; provides audience with too much information (unable to adequately prioritize)
E3—Uses well-founded, overarching guidelines or principles to objectively compare and choose among alternative solutions; provides reasonable and substantive justification for assumptions and choices in light of other options*
F3—Focuses on pragmatic issues in implementation plans; provides appropriate information and motivation, prioritized for the setting and audience*
E4—Articulates how a systematic process of critical inquiry was used to build solution; identifies how analysis and criteria can be refined, leading to better solutions or greater confidence over time
F4—Implementation plans address current as well as long-term issues; provides appropriate information and motivation, prioritized for the setting and audience, to engage others over time
Step 4: ENVISION G—Acknowledge and monitor solution limitations through next steps H—Overall approach to the problem
G0—Does not acknowledge significant limitations beyond temporary uncertainty; next steps articulated as finding the “right” answer (often by experts)
H0—Proceeds as if goal is to find the single, "correct" answer
G1—Acknowledges at least one limitation or reason for significant and enduring uncertainty; if prompted, next steps generally address gathering more information
H1—Proceeds as if goal is to stack up evidence and information to support own conclusion
G2—Articulates connections among underlying contributors to limitations; articulates next steps as gathering more information and looking at problem more complexly and/or thoroughly
H2—Proceeds as if goal is to establish an unbiased, balanced view of evidence and information from different points of view
G3—Adequately describes relative importance of solution limitations when compared to other viable options; next steps pragmatic with focus on efficiently GATHERING more information to address significant limitations over time
H3—Proceeds as if goal is to come to a well-founded conclusion based on objective consideration of priorities across viable alternatives
G4—Identifies limitations as in G3; as next steps, suggests viable processes for strategically GENERATING new information to aid in addressing significant limitations over time*
H4—Proceeds as if goal is to strategically construct knowledge, to move toward better conclusions or greater confidence in conclusions as the problem is addressed over time*
© 2006, Susan K. Wolcott. All rights reserved. Materials herein may be reproduced within the context of educational practice or classroom education, provided that reproduced materials are not in any way directly offered for sale or profit. Please cite this source: Wolcott, S. K. (February 9, 2006). Steps for Better Thinking Rubric [On-line]. Available: http://www.WolcottLynch.com. Based in part on information from Reflective Judgment Scoring Manual With Examples (1985/1996) by K. S. Kitchener & P. M. King. Grounded in dynamic skill theory (Fischer & Bidell, 1998). * Shaded cells most closely related to "stair step" model. Performance descriptions to the left of a shaded cell characterize skill weaknesses. Performance descriptions to the right of a shaded cell characterize skill strengths.