The DQP line from proficiencies to assignments to records

Cliff Adelman, Institute for Higher Education Policy

March 28, 2014

What are we going to talk about?• Language, faculty, IR, and registrars• How to write proficiency statements under which

student performance can be judged.• Language signals: statement type, voice, diction level.• The line working backwards from assignments, and the

“Ah, hah!” moment for faculty.• The challenge of recording proficiencies.• The greater challenge of reporting student proficiencies.

The form of proficiency statements, what’s out:• “Ability.” You don’t know a student has an “ability” to do

something until they do it. “Ability” and “capacity” are white noise.

• “Awareness.” At best, it’s peripheral consciousness, and you don’t award degrees based on consciousness.

• “Appreciation.” Tell us what a student does when they “appreciate” something! Even music conservatories don’t use the term.

• “Critical thinking,” the championship mush phrase of the millennium. Describe the cognitive operations of “critical thinking” and you have true assignment-oriented lead-in verbs! You can drop the foggy short-hand now!

The form of proficiency statements: no lead verbs, no assignments! So what’s in, Part I?• Delineating: categorize, characterize, classify,

define, describe, determine, frame, identify, prioritize, specify

• Explicating: articulate, clarify, explain, illustrate, interpret, outline, translate

• Examining: analyze, compare, contrast, differentiate, distinguish, extract, formulate, map

• Combining: assimilate, consolidate, connect, integrate, link, synthesize, summarize

I think you begin to see what I mean, so. . .

Want some more cognitive action verbs for proficiency statements? Try the following categories:• Inquiring: experiment, explore, hypothesize,

investigate, research• Making: build, compose, construct, craft, create,

design, develop, generate, model, shape, simulate• Operating: administer, control, coordinate, engage,

lead, maintain, navigate, optimize, plan, undertake• Converging: collaborate, contribute, interact,

negotiate, participate• Valuating: audit, appraise, assess, evaluate, judgeAnd so forth. Some of these are stronger prompts than

others; some could appear in two categories.

A lead into noun problems, too: Why is language important in the proficiency world?• It signals statement type (competence, discrete outcome,

metacompetence)• It signals status of the proficiency (possession, development,

goal)• Voice (declarative, imperative, subjunctive) signals the student

whether fulfilling or exceeding the performance specified is assumed, required, or simply desired. This is a matter of intentionality.

• Diction level (on a concrete/abstract continuum) is a camera lens setting on the breadth, tractability, and accessibility of the statement’s subject.

All this leads to our biggest noun problems: “knowledge” and “understanding”• The terms are too often used as synonyms. No deal!• Bloom et al’s Taxonomy discards “understanding” and

replaces it with “comprehension,” a combination of translation, extrapolation, and interpretation, to which we might add description, inference, visualization and testing.

• “Knowledge” is too often presented as an assumption, i.e. that everyone knows what it is. We feel too free to speak of “advanced knowledge” in the face of the question, “What might ‘not so advanced’ mean?”

The principal problem with “knowledge” is that our language assumes it is something possessed, a status and not an event. Really?• It comes out as a tautology, as in “I have

possession of what I possess,” or• in an example from a UK benchmarking statement

for accounting, “basic knowledge and understanding is characterized by knowledge of a topic” I am not kidding!

• “Knowledge,” though, leads to specifications, and we understand this best in the disciplines, and see it in Tuning.

Let’s mark, then, what is in “Tuning,” a process now present on 6 continents• “Tuning” is a ground-up faculty driven process to create

templates of reference points for student learning outcomes in the disciplines.

• Then to go on and write student learning outcome statements and their accompanying assessments referenced to the template.

• And the disciplines are the locus for both outcome statements and assignments, a natural by-product of where faculty are located, where they are trained, and how they self-identify.

• The result is “convergence,” not carbon copies.

Example of a “reference point” in business administration: the Firm• Not merely “the firm,” but the firm as a “value-chain” from

procurement to customer service• And with stops along the way in production (goods or

services), logistics, accounting, marketing, etc.• In the European Tuning project, representatives from

universities in 15 countries speaking 11 languages agreed to this.

• Did that mean they all went home and taught “the firm” the same manner with the same weightings? Hell, no! But it meant they would use the “value-chain” concept in their own sweet ways.

“Tuning USA” first followed this path in Indiana, Minnesota, and Utah• You need a state system to pull it off---well, not always.• Unlike the Euros & others, we include the associate’s

degree, hence community college faculty• Also, unlike the Euros & others, we had students on

every disciplinary team• Each state picked 2 or 3 disciplines; Indiana and Utah

independently selected history.• Texas later joined with 4 engineering fields, and

Kentucky came in with 5 disciplines.

Back to the language of proficiency statements and “knowledge”While “knowing” is a verb, the position of “knowledge” in

Tuning is an irrevocable noun; the reference points specify : “knowledge” of a disciplinary what.

And that’s where the lists begin: legal regulations, error analysis, transport phenomena, major wars, poetic forms, auditing principles, and on and on.

Given the lists, the governing verb goes beyond possession to presentation, e.g. demonstrate, display, perform, etc. “Knowledge” becomes an event, not a status.

When you put together all the elements described above. . .• You have the tools and forms for writing learning

outcome statements in your field.• What you don’t have yet, are natural forms of eliciting

from students the behaviors that allow you to validate whether they have demonstrated proficiencies.

• But those natural forms are staring you in the face because you, as faculty, create them every week. They are called assignments. And they need no more than tweaking to produce a clear connection between proficiency objective and its validation.

You don’t have to look far for generic cases, e.g. • From the DQP’s bachelor’s-level for quantitative fluency:

“The student translates verbal problems into mathematical algorithms so as to construct valid arguments using the accepted symbolism of mathematical reasoning and presents the resulting calculations, estimates, risk analyses or quantitative evaluations . . .”

• You have a hand-out that was a real-world advertisement, a serendipitous find that we tweaked to flow from the above proficiency statement. Let’s talk about it, and see if we hit the “Ah, ha! moment”

And at the associate’s level • For integrative knowledge, something the DQP does not leave

just for bachelor’s and master’s programs: “Describes a key debate relevant to [a core field]. . . And shows how concepts from core field can be used to address the . …debate.”

• And then the assignment (published in DQP 2.0):

Prepare an exhibit of not more than five discrete 2-dimensional pieces illustrating the range of chaos in color, drawing on at least two of the major color theory sources, e.g. Goethe, Kandinsky, Chevruel, in a 3-5 page catalogue of your exhibit. . .

Well, it’s not a perfect match, but close. There is a real debate on color.

Similar problems, presented as assignments, test questions, field projects, etc. can be developed and offered in many fields• Not all fields, to be sure, but you need enough of

them to assert that MKCC has a portfolio of prompts to validate this type of quantitative proficiency.

• And what you do for quantitative proficiency and integrative knowledge you can do for all the others in the DQP, or your version of a DQP.

• And of course an assignment cannot cover 3 or 4 proficiency territories.

Challenges of putting together a set of touchstone assignments• A review committee to determine validities• Decision rules on distribution of qualifying

assignments across disciplines and levels• Protection against concentrations of qualifying

assignments in two or three curricular areas• Recognition that assignments change, and establishing

a process for continued change and review• Making sure that individual faculty will cover but a

limited number of proficiencies

Let’s be optimistic now, and assume• The ASU faculty senate has endorsed its own version of the

DQP; there are 36 proficiencies• The Review Committee has marked 288 assignments as DQP

validations, with at least 5 for each proficiency, and scattered across 50 departments/programs

• A separate student tracking analysis group has examined the records of 1,000 students to make sure each of them would have encountered all assignment-validating proficiencies in the course of their undergraduate careers (talk about a task!). And confirmed that such is the case.

WOW!

What’s missing? And What Does One Do Next?• Grades are missing. The DQP process does

not interfere with faculty judgment of student performance. You can set alternative levels of judgment, but not issue the judgments themselves: that’s a faculty prerogative.

• Next step is the establishment of a record system, its inputs, and its summative Degree Audit conclusion. So. . .

Consider the following for record-keeping• Every student has an electronic template with the 36

proficiencies in boxes• Faculty responsible for validated assignments enter

judgments (if binary, only checked if satisfied; if multiple-leveled, then level) with a secure digital entry

• You decide whether the student can receive multiple checks in the same box. I prefer that if faculty see the box previously checked, they don’t have to duplicate the judgment.

• Credits continue, but in a separate account.

So when do you get a degree?• All 36 boxes checked• Credit requirements met• Residency and recency requirements met• No financial or disciplinary holds• “Opt-out” degree award policy (ask me!)• Registrar validates the whole; certifies the

Degree Audit.

Does this look like DQP “on the ground”?• It’s not on auto-pilot after established• It’s missing provisions for transfers-in from institutions

that have nothing comparable (granted, ASU transfer-in percentage is low).

• It’s missing key inclusion items for adjunct faculty.• It opens potential conflicts from faculty whose

assignments are not included as validating.• It does not answer the question of what ASU could show

to external parties other than the 36 proficiencies. Maybe retired assignments.

But of all the obligations we have in higher education, our primary responsibility is to students, and this DQP process is just that! • The generation of knowledge and local community

economic stimulus can go on elsewhere, but, in our ball game

• the core operating noun is “student”; the operating verbs are those of specific actions by students; the operating results are criteria for the award of degrees to students. Get the idea?

• There still is dirt here; there are still rough edges. You can clean it up, sand it, and make it work!