operationalizing and assessing the entrepreneurial mindset: a rubric
TRANSCRIPT
Volume 2, Number 2, Special Issue 2011
*Associate Professor of Marketing, Pharmaceutical Business Program Coordinator
James F. Dicke College of Business Administration, Ohio Northern University
57
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Robert E. Kleine, III * and John-David Yoder
Abstract – A process for developing and assessing programming -- curricular,
co-curricular, or extracurricular -- intended to cultivate the entrepreneurial
mindset in engineering students is described. Steps of the process are to 1) define
learning-outcomes; 2) create a rubric for each learning-outcome; 3) develop
activities that generate student work; 4) collect student work; 5) conduct direct
assessment of student work by applying the rubrics to student work generated by
the activities; 6) interpret and report assessment results; and 7) modify learning-
outcomes, rubrics, activities or other program elements in response to
assessment results.
1. Introduction and Background
Stakeholders at educational institutions worldwide (e.g., students, parents, boards of trustees,
state legislators, accrediting agencies, granting agencies and foundations, employers) increasingly
expect evidence that an institution’s educational programs are effective (Arum and Roska, 2011).
The proliferation of disruptive new business models for delivering educational experiences,
enabled by advances in information technologies, increases the pressure on institutions to
document their ability to develop and deliver programs that achieve claimed outcomes
(Christensen et al., 2008). Current economic conditions, specifically declining tuition and
investment revenues experienced by many institutions, further increases the pressure to document
the degree to which programs are effective. These forces apply equally to engineering
entrepreneurial education programs.
Effective programs, courses, or class activity designs require clarity on the learning-outcomes to
achieve and on the tools that will be used to assess them (Fink, 2003; Smith, 2008). Learning-
outcomes are essential as they guide creation of programs, courses, class activities, and/or extra-
or co-curricular activities. Learning-outcomes also provide the benchmarks against which student
achievement is measured. Such assessment of student performance reveals program effectiveness
and guides program improvement efforts. Designers of engineering entrepreneurship programs
must specify carefully the learning-outcomes that motivate their programs. Business creation
(Bhide, 2000), if specified as a program outcome, will imply a program very different from a
program designed to cultivate the entrepreneurial mindset, which emphasizes cognitive or
attitudinal outcomes (Riffe et al., 2010). This article grew out of work with the Kern
Entrepreneurship Education Network (KEEN). The KEEN program, an initiative of the Kern
Family Foundation, is a network of schools working collaboratively to cultivate entrepreneurial
engineers characterized by an entrepreneurial mindset (Kriewall and Mekemson, 2010).
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 58
Designing and implementing engineering education to cultivate students’ entrepreneurial mindset
requires clarity on three key issues:
1. What is the entrepreneurial mindset (EM)? The EM must be defined in a way that is
appropriate for development of specific learning-outcomes.
2. How can the entrepreneurial mindset be operationalized in a way that is appropriate for
guiding the design of educational activities intended to cultivate the entrepreneurial
mindset? The EM learning-outcomes must be expressed in a way useful for cultivating
EM in students.
3. How can those activities be assessed to determine whether or not they have had the
intended effects upon students’ entrepreneurial mindsets? We must be able to identify
what activities effectively cultivate the EM and those that do not.
Effectiveness insights are essential for enabling continuous program improvement. This article
leverages work completed by KEEN to define the entrepreneurial mindset to propose a learning-
outcome driven program development and assessment process model. The model is a framework
intended to guide anyone involved in cultivating the EM in students, whether an individual
faculty member, program coordinator, dean, or the director of a multi-school consortium (such as
KEEN).
The next section provides a brief overview of the proposed model of the learning-outcome driven
program development and assessment process. Subsequent sections unpack model components.
We conclude with suggestions for implementing the proposed model so as to maximize
cultivation of the entrepreneurial mindset among our students.
2. An EM Program Development and Assessment Process Model
A normative model of the learning-outcome driven program development and assessment process
for cultivating the entrepreneurial mindset is depicted in Figure 1. This model reflects inspiration
from several sources, most specifically Fink’s model (2003) for creating significant learning
experiences and the Association of the American Colleges and Universities (AAC&U) initiative
to develop normative models to guide assessment of undergraduate general education
(Association of American Colleges and Universities, 2011). The process is inherently adaptive
and recursive. Experiences applying each component of the model may precipitate modifications
intended to more closely align the system with desired objectives. The following sections
describe each of the model’s six key elements.
2.1. Learning Outcomes
Learning outcomes that reflect the entrepreneurial mindset are the foundation to this approach.
Learning outcomes are typically developed by each institution (general education learning
outcomes, for example) or units within an institution such as colleges or departments (e.g.,
learning outcomes for a major or for individual classes; see Otter (1995) for a nice overview of
learning outcomes). The process through which the EM learning outcomes were developed is
unique in that the KEEN schools collaborated to develop a set to apply across institutions.
Although a full chronology of the process employed by the KEEN schools is beyond the scope of
this paper, it is important to note that this inter-institutional approach to development of learning
outcomes is uncommon. The AAC&U LEAP initiative, which is developing a common set of
general education learning outcomes is an exemplary exception. The EM learning outcomes were
developed through a yearlong, multi-step process. The process included representatives from the
various KEEN institutions, all private institutions with ABET accredited engineering programs,
who met on several occasions with the goal of achieving consensus on a set of learning outcomes.
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 59
The resulting suite of seven EM learning outcomes defines the Entrepreneurial Mindset for the
KEEN schools (see Table 1).
Figure 1. A Model of the Entrepreneurial Mindset Learning-outcome Driven Program
Development and Assessment Process.
Table 1. Entrepreneurial Mindset Learning-outcomes Developed by the Kern Entrepreneurship
Education Network
A student should be able to:
1. Effectively collaborate in a team setting
2. Apply critical and creative thinking to ambiguous problems
3. Construct and effectively communicate a customer-appropriate value proposition
4. Persist through and learn from failure.
5. Effectively manage projects through appropriate commercialization or final delivery
process.
6. Demonstrate voluntary social responsibility.
7. Relate personal liberties and free enterprise to entrepreneurship.
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 60
This paper focuses on the role of learning outcomes in the context of developing and assessing
programs to cultivate the entrepreneurial mindset. Explication of how to construct these is beyond
the scope of this paper. For purposes of EM program development and assessment, learning
outcome statements provide direction to development and assessment of programming intended
to cultivate specific outcomes in students. When used in an inter-institutional context, as with the
KEEN schools, learning outcomes are also a vehicle for focusing the collective energies of
disparate institutions toward a cultivating a common set of student abilities.
It is important to realize that the suite of seven learning outcomes applies to the totality of a
program. An individual activity, or even a single course, cannot cultivate all of the abilities
implied by the EM learning- outcomes. A specific activity, such as an in-class egg case design
exercise, strives to cultivate a focused subset of the EM learning outcomes. The myriad EM
cultivating activities deployed by a program, in aggregate, yield the total impact of a program.
One challenge with learning- outcomes, even those that are well crafted , is that they are often
subject to multiple interpretations. Different interpretations of learning outcomes by individual
faculty members or by disparate institutions induce variance in what abilities students develop
with regard to a particular learning outcome. Rubrics are a tool commonly used to enhance
uniformity of understanding of learning outcomes.
2.2. Rubrics to Operationalize the Entrepreneurial Mindset
A rubric is an explicit set of criteria used for assessing a particular type of work or performance
(The TLT Group, 2011). Rubrics serve several vital functions. This paper emphasizes the utility
of rubrics for EM program development and assessment. Rubrics can serve other important
functions. For example, rubrics can function as a communication tool among faculty within an
institution or across institutions, as was mentioned above. When shared with students, rubrics
afford an efficient tool for conveying performance expectations. Applied in the assessment
process, rubrics serve as measuring devices for systematizing the assessment of student work.
Rubrics are especially helpful when, as is often the case, the learning outcome has a subjective
component. A rubric is a tool that helps objectify the assessment of subjective phenomena. A
rubric provides a set of criteria developed to operationalize a learning outcome.
The unit of application must be identified when developing learning outcomes and hence rubrics.
The entrepreneurial mindset rubrics (reproduced in the Appendix) were developed for the purpose
of assessing the work of an individual, as opposed to groups of students. The rubric is applied to a
unit of work that can be identified as having been produced by a specific student. Individual
student performance on a group activity can be assessed if the individual’s contribution is distinct
and separable from the work of the other group members. For example, effective verbal
communication for an individual could be determined from their part of a group presentation if
there is a portion of the presentation during which the individual is the only group member
speaking. In contrast, a group-produced written report would not be an appropriate artifact for
assessing written communication effectiveness for individual students.
Assessment rubrics should be shared with students at the time relevant activities are assigned.
The rubrics thus afford students a useful guide for how to approach the activity. It is also
productive to instruct students to self-assess their work with the relevant rubrics. The self-
assessment might afford the foundation for a reflection paper. The self-assessment might also be
directed toward closing the self-identified weaknesses in the work.
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 61
2.2.1. A Recipe for Rubric Creation
A general recipe for constructing an assessment rubric is as follows:
a) Dimensionalize the learning outcome according to the distinct student tasks or abilities
specified in the learning outcome. Rubrics are commonly summarized in matrix form. The
dimensions typically form the rows of a rubric. This step can be tricky when a learning outcome
includes a fundamental ability that is an implied or assumed but not explicitly stated. For
example, consider the learning outcomes ―Persist through and learn from failure.‖ This learning
outcome contains two explicit dimensions: persist through failure and learn from failure. Implicit
in this learning-outcome statement is the ability for a student to recognize when a failure has
occurred. An individual must recognize that a failure has occurred before they find evidence of
persistence and learning. Consequently, the rubric for this learning outcome has three dimensions:
―failure recognition,‖ ―persist through failure,‖ and ―learn from failure‖ (see the Appendix). For
this reason, the individual(s) creating a rubric must be proficient in the domain addressed by the
rubric.
b) Determine the number of performance levels. Performance levels typically form the columns
of a rubric matrix. Performance levels delineate qualitatively distinct levels of student
performance. The performance levels progress monotonically and typically range from absence of
the desired ability (at the low end) to proficiency (at the high end). The number of intermediate
performance levels included must strike a balance between making distinctions that capture
incremental variation in qualitatively distinct levels of student performance and discriminating
the results by all who will use the rubric. In general, ease of use declines as the number of
performance levels increases. Although we have not found research on this topic, four levels of
performance appear to strike the balance between capturing essential performance variance and
ease of use.
Four performance levels are used for the entrepreneurial mindset learning-outcome rubrics:
Does not meet expectations, Developing, Meets Expectations, and Proficient.
―Does not meet expectations‖ characterizes a student performance that does not display
any of the desired target activity.
―Developing‖ characterizes a student performance that evidences some of the desired
target activity.
―Meets expectations‖ designates student performance that evidences the minimal level of
ability expected.
―Proficient‖ designates student performance that exceeds ―meets expectations‖ and
evidences mastery of the target activity.
c) Identify student abilities that reflect performance characteristic of each dimension. It is useful
to ask the question: What would provide evidence that a student has mastered this dimension?
The resulting answer can form the foundation for describing the ―proficient‖ level of the
dimension. For example, proficiency at the ―failure recognition‖ dimension was determined to be
indicated by two activities: recognition that a failure has occurred and the ability to properly
identify (categorize) the failure. Bloom’s taxonomy (Bloom et al., 1956) is a useful aid for
articulating the desired behavior characteristic of proficiency.
The act of describing what constitutes ―proficient‖ behavior may reveal that a dimension is very
complex and should be disaggregated into sub-dimensions to fully capture the breadth of the
domain encompassed by the learning outcome. The Effective Communication rubric reflects the
complexity of the learning outcome. This rubric includes two levels of sub division. First, it
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 62
differentiates three types of communication—written, oral, and visual--on recognition that each is
a unique behavioral domain. Second, written and oral share the ―overall organization‖
dimension—the behaviors that reflect that behavior differ by communication modality. The rubric
descriptors reflect these differences.
Each cell in the rubric matrix presents a description of the student behavior characteristics of the
specific ability level along that dimension. The ability levels increase monotonically. Each ability
level subsumes the performance abilities of the levels below it. Pragmatically, it is often easiest to
write the description of the highest-level indicator and then back fill the lower levels. For
example, proficiency of failure recognition is characterized by evidence of three conceptually
related events: failure recognition, failure identification, and the ability to think critically about
the consequences of the failure. Conceptually, recognition that a failure has occurred must occur
before identification and apprehension of the consequences. Put differently, if a student is
unaware that a failure has occurred, none of the subsequent behaviors can be present.
Consequently, the descriptors progress from the following behaviors: not evident; evident at a
low level (recognition that failure occurred); evident at a qualitatively distinct higher level
(recognition that failure occurred and proper identification of the failure); evident at a level that
reflects mastery of the dimension (recognition that failure has occurred, correctly identifying the
failure, and explaining the consequences of the failure).
The specific student behaviors that reflect each level will depend on the context in which they are
applied. For example, evidence of failure recognition in the context of a freshman engineering
activity, in which team constructed bridges that were stress-tested, will differ from the evidence
of failure recognition if the student was analyzing a business model.
Rubric construction can be expedited or avoided altogether by first searching for existing rubrics
constructed by others (rubistar.com and tltgroup.org provide inventories of rubrics and tools for
rubric construction). When developing rubrics for the EM learning outcomes, we leveraged the
fact that several of the learning outcomes overlap with general education learning outcomes
developed by the AAC&U and Ohio Northern University (ONU). The AAC&U have published a
set of general education learning outcomes and rubrics with the intent that institutions use them as
meta-rubrics to inspire institution- or program-specific rubrics. The AAC&U rubrics were
developed through the assistance of faculty from institutions around the world. The rubrics
developed by Ohio Northern University (2009), some of which were inspired by the AAC&U
efforts, were created to support ONU’s new general education program. ONU rubrics were
constructed by teams comprised of ONU faculty with relevant expertise via a modified Delphi
technique facilitated by EduMetry (http://edumetry.com/), a consultancy that specializes in higher
education program assessment. The use of teams, comprised of faculty from across the ONU
campus, was employed to increase the likelihood that the resulting rubrics, as they result from
faculty engagement, would be more readily accepted and adopted across disciplines.
2.2.2. EM Rubrics: An Overview
A brief overview of the rubric developed for each EM learning outcome follows. It is important
to keep in mind that rubrics are typically part of a suite of tools that faculty, programs, or
foundations might use to assess the effectiveness of a program. Also, these rubrics are intended to
provide a starting point. It is expected that the rubrics will be modified as application data
becomes available. These overviews are intended to provide background on the origins and logic
underlying each of the EM learning-outcome rubrics. The rubrics themselves are presented in the
Appendix.
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 63
Effectively collaborating in a team setting – This rubric built on the foundation afforded by the
AAC&U teamwork rubric, specifically, the AAC&U rubric’s five dimensions: contributes to
team meetings, facilitates the contributions of team members, the individual’s contributions
outside of team meetings, and fosters constructive team climate. Working in collaboration with
two ONU management faculty, the target behaviors were adjusted in ways that better align with
salient target attributes based on their understanding of the teamwork literature and experience
with senior capstone groups. Emphasis was placed on ―soft skill‖ attributes that are sufficiently
objective such that their presence or absence can be reliably detected. The ―contributes to team
meetings‖ dimension captures the degree to which an individual effectively engages during team
meetings. ―Facilitates the contributions of team members‖ focuses on the degree to which team
members respect and enable contributions by other team members. ―Individual’s contribution
outside of team meetings‖ captures the degree to which an individual completes his/her team
assignments in accordance with the agreed upon time schedule and assures that the work done is
of a quality that advances the project effort toward its goals. ―Fosters constructive team climate‖
focuses on the degree to which an individual respects, motivates, encourages, and supports team
members. Conflict is common in group activities. How individuals handle conflicts can determine
whether a group becomes dysfunctional or succeeds and proceeds forward. The ―responds to
conflict‖ dimension captures the degree to which an individual engages and resolves conflict in
an appropriate way.
Applying critical and creative thinking to ambiguous problems – Critical and creative thinking
are considered by many to be the holy grail of learning outcomes; cultivating critical and creative
thinking is regarded by many as the primary purpose of an undergraduate college education
(Arum and Roska, 2011). The ability to think critically and creatively is fundamental to
entrepreneurial activity. Ambiguous problems refer to the ill-structured nature of the situations to
which the entrepreneurial mindset is applied. Ill-structured problems contrast with well-structured
problems. Well-structured problems have an unambiguously correct solution. For example, ―what
is the load bearing capacity of a bridge made from carbon fiber of a specified density and
dimensionality?‖ In contrast, ―what is the best way to study for my statistics exam?‖ is an ill-
structured problem. There are myriad possible solutions. The challenge is to apply critical and
creative thinking to identify the solutions that are most appropriate given the constraints and
resources available.
The rubric presented here was developed by a team of ONU faculty for application with ONU’s
general education program. The learning outcome is elaborated as the following: students are able
to correctly identify the underlying problems or issues in both theoretical and practical realms;
apply appropriate analytical and creative skills to develop feasible alternatives while considering
multiple perspectives; and provide creative and logical solutions. The ONU team process was
seeded by two AAC&U rubrics: critical thinking and creative thinking. The resulting rubric has
three sections: problem identification, research and analysis, and solution development.
Constructing and effectively communicating a customer-appropriate value proposition –
Innovation and entrepreneurial activity are complementary yet distinct activities. Entrepreneurial
activity concerns innovation harnessed in the service of providing value to an identified market.
This learning outcome has two distinct components: constructing a customer-appropriate value
proposition and the ability to effectively communicate that value proposition to key
constituencies (e.g., end users, purchasers, investors).
A search failed to yield a rubric appropriate for the customer-appropriate value proposition
learning outcome. SRI International’s NABC model—Need, Approach, Benefits relative to costs,
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 64
and Competition—for formulating a value proposition (Carlson and Wilmont, 2006) was thus
adopted to operationalize this learning outcome for three reasons. 1) The NABC model has a
proven record of success (Carlson and Wilmont, 2006). 2), The NABC model has proven useful
as a pedagogical aid in Kleine’s Product Development class. NABC provides a framework
students grasp readily and can apply productively when assessing and expressing a value
proposition. 3) The KEEN community, the ecosystem in which the rubric is to be deployed, is
familiar with the SRI model. This familiarity is conducive to accelerating adoption of the rubric
across the KEEN schools.
The rubric operationalizes the key dimensions of the NABC formulation:
Is an important customer and market need addressed?
Is the proposed approach meaningfully unique or distinctive relative to alternatives?
Does the proposed approach offer distinctive benefits relative to costs?
How do the benefits/costs of the proposed solution compare to existing alternatives?
Are the benefits/costs of the proposed solution quantitatively superior to existing
alternatives?
Are key constituencies identified and an appropriate variation of the value proposition
formulated for each? Key constituencies might be prospective investors, channel partners,
and end-users. The value proposition appropriate for an end-user would not be
appropriate for a channel partner or a prospective investor.
This rubric implicitly subsumes myriad sub-activities necessary to effectively support each level.
For example, the ability to identify key constituencies and identify and quantify market needs
presumes market research and analysis are conducted. The rubric is crafted such that a
comprehensive business plan is not mandatory. The rubric also does not presume that a working
prototype is crafted, but rather the emphasis is on making the case for the potential of an idea.
Effective communication is one of ONU’s general education learning outcomes. In the interest of
efficiency, ONU’s rubric for effective communication general was adopted. This also illustrates
that the rubrics proposed are modular. If an institution has effective communication as a general
education or other learning-outcome and has a rubric developed for assessing it, that rubric could
be adopted to leverage existing infrastructure on campus. ONU’s effective communication
learning outcome has three major components: effective written communication, effective verbal
communication, and effective nonverbal communication.
Persisting through and learning from failure – An individual’s ability to overcome adversity, to
stick with a problem until an effective solution is found, to ―fail-forward‖ effectively (Maxwell,
2000), is characteristic of ―growth mindset‖ individuals (Dweck, 2000). Others, such as Dwek’s
―fixed mindset‖ individuals are less likely to try again upon experiencing failure. Dwek’s
research shows that ―fixed mindset‖ individuals can be taught how to approach situations using a
growth mindset approach. This research suggests that the ability of an individual to persist
through and learn from failure is a skill and not a personality characteristic inherent in
individuals. Consequently, individuals can cultivate their ability to negotiate failure.
This learning outcome has three fundamental dimensions that build on one another: failure
recognition, persist through failure, and learn from failure. The ability to recognize that a failure
has occurred is essential as it sets the stage for corrective action. Consequently, the first row of
the rubric captures the ability to identify, categorize, and apprehend the consequences of the
identified failure. Implicit in this formulation is the possibility that a recognized failure is
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 65
misidentified. Improper identification or categorization of the failure will have implications for
the individual’s ability to effectively mitigate the failure.
Identification that a failure has occurred sets the stage for figuring out how to deal with
the failure. Persisting through failure is operationalized in terms of an individual’s ability to
identify and articulate a course of action that could mitigate, reduce or eliminate the
consequences of the identified failure.
Effectively managing projects through appropriate commercialization or final delivery process –
This learning outcome recognizes that a good customer-appropriate value proposition must be
translated into reality. That translation requires understanding of the process through which an
idea is transformed into a valuable entity. The rubric has four primary dimensions: 1) project
level, 2) task level, 3) evaluation and 4) deliverables.
The project level dimension captures activities relevant to overall project management: goal
setting, steps to achieve goals, optimal use of resources to achieve those goals, and the ability to
create and maintain a budget. This rubric was developed with the recognition that different
programs teach different process models. For example, some programs emphasize the stage-gate
model (Cooper, 2011) for managing the product development process. Consequently, students
would be expected to evidence understanding of the stage-gate model in how they approach
project management.
Embedded within the overall structure of a project are the myriad tasks that must be identified,
defined, sequenced and implemented in a way that advances the project toward its ultimate goals.
These task-level project activities include defining and sequencing appropriate tasks, calculating
the time a task will require, executing the task, and managing the task process, which may require
delegating and/or coordinating with team members.
Evaluation captures the ability to effectively assess progress toward the identified goal(s). This
includes the ability to articulate procedures to use to assess project effectiveness into the future.
Deliverables, the materials that must be created and supplied to designated constituencies by an
identified time form the final dimension. This dimension reflects that projects typically require
generation of materials that provide evidence of project activity. The specific materials to be
delivered will depend on the project and its goals. For example, one course may have creation of
a development plan as its key deliverable. Another course may entail a working prototype with
supporting documentation as expected deliverable materials.
Demonstrating voluntary social responsibility – This learning outcome parallels ONU’s general
education learning outcome for ―informed and ethical responses to personal, civic and global
needs. ― These learning-outcomes parallel the AAC&U ―Civic Engagement‖ learning outcome.
However, the KEEN learning outcome is not isomorphic with these two learning outcomes.
Accordingly, the rubric (see the Appendix) leverages those foundations, yet embraces the
uniqueness of the KEEN learning outcome. The target behaviors are that students display an
understanding of personal, civic and social needs; identify underlying ethical issues in specific
situations; and make informed ethical responses to those needs. This parsing strives to embrace
the spirit of the KEEN learning outcome while maintaining as much overlap as is possible with
the ONU and AAC&U general education rubrics so as to maximize utility to schools that may
apply it.
The rubric has three dimensions that operationalize the three parts of the learning outcome. The
first dimension concerns the degree to which the individual evidences an understanding of needs
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 66
that are grounded in data as opposed to personal opinion. The second dimension concerns an
individual’s ability to discern the ethical issues that arise from social needs. The third dimension
concerns the individual’s ability to respond to identify needs in an appropriate way by applying
appropriate analytical frameworks.
Relating personal liberties and free enterprise to entrepreneurship – This learning outcome
highlights individuals’ understanding of personal liberties and free enterprise and how each is
reflected in and enabled by entrepreneurship (Schramm, 2006). As such, this learning outcome
focuses on the intersection of values, government regulation, and individual initiative directed
toward creating and delivering value. The rubric’s three dimensions flow naturally from the
learning outcome. Entrepreneurship, the first dimension, focuses on the student’s understanding
of entrepreneurial behavior and attributes of successful entrepreneurial behavior. The second
dimension taps the student’s ability to provide evidence of a connection between personal
liberties and entrepreneurship. The third dimension focuses on individuals understanding of free
enterprise and entrepreneurial behavior.
2.3. Developing curricular, co-curricular, and extra-curricular activities, and programs that
cultivate the EM
An outcomes-based approach to program development implies that learning outcomes are
simultaneously the beginning and the result of the education cycle. Rubrics, by providing an
elaboration of each learning outcome, afford a tool useful for aiding the design of educational
activities. ―Activity‖ is used here as a generic term for curricular, extra-curricular and/or co-
curricular activities intended to further a learning outcome.
Student abilities are typically cultivated via modular units such as majors, minors, courses, extra-
curricular activities such as student organizations, and co-curricular activities such as idea pitch
competitions. The smaller the modular activity unit, the more specific the learning outcomes the
activity will develop. In the case of the KEEN program, for example, all seven of the
entrepreneurial mindset learning-outcomes apply. Program-level design involves asking the
question: how will we cultivate the abilities of our students with the goal of them achieving
proficiency? As with all unstructured problems, there is no single best answer. What is a best
approach for one school may not be appropriate for another school.
2.3.1. The Program Matrix
A Program Matrix is a visual tool useful for guiding program development. Form a Program
Matrix by expressing the learning outcomes as columns. Express the various EM cultivating
activities available on a campus as the rows. Place a check mark or other symbol in the table cells
to designate the specific EM learning outcome(s) addressed by a particular activity. The result is a
convenient visual summary of an institution’s EM cultivating initiatives. Review the summary
relative to the objectives established for your institution’s EM initiative. Identify learning
outcomes lacking coverage. These voids should be addressed. Similarly, review the summary to
identify over-emphasized learning outcomes. Over emphasized areas could be scaled back. The
resources freed up (e.g., class time) can be reallocated to develop activities that cultivate the
underserved learning outcomes.
As a first pass, the matrix may include only the seven learning outcomes. As the program design
becomes better specified, expand the learning outcomes into their constituent dimensions and
rubric rows. Review the resulting Program Matrix to identify voids. The goal is for each learning
outcome to map onto at least one activity.
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 67
2.3.2. Aligning Activities with Learning Outcomes, Rubrics and Rows
Activities should be closely aligned with the EM learning outcomes and rubrics. Activity/rubric
alignment requires addressing two questions: 1) What learning outcome(s) will this activity
cultivate? This step identifies the relevant rubrics. 2) What row(s) of the relevant rubrics apply?
Typically, an activity will cultivate one or two of the EM learning outcomes and specific rows of
the relevant rubrics. A single assignment created for a particular course might focus on parts of
one or more learning outcomes. For example, an activity that requires the student to develop and
deliver an elevator pitch might be used to develop a student’s ability to construct a customer
appropriate value proposition and her oral communication skills. When developing the activity,
care must be taken to align the activity task with the behaviors emphasized by the relevant EM
rubric rows.
Rarely will a single activity suffice for bringing students to the target proficiency level for a
specific learning outcome. One activity will generally impact incrementally the degree to which
participating students are proficient. In many cases, multiple activities that cultivate a single
ability will be required for students to achieve proficiency. For example, critical and creative
thinking might be developed throughout a student’s undergraduate program. A single activity
might simultaneously cultivate multiple learning outcomes. For example, a group activity focused
on developing criteria for screening ideas, may simultaneously cultivate critical thinking to solve
ambiguous problems, team collaboration skills, and an element of constructing a customer-
appropriate value proposition. It is useful to prioritize the intended learning. It is also possible
that an activity will cultivate specific parts of a learning outcome. For example, an activity that
requires students to prepare a slide deck to support defense of a selected alternative might be
identified as an activity that cultivates visual communication skills.
The rubrics for the learning outcomes (and sub-parts) to be emphasized in a particular activity
provide useful guidance on the types of work the students might do to cultivate the outcome.
They provide guidance on the specific skills that should be emphasized or that may be developed
prior to commencing an activity. The rubrics can also be used as a way to check activities to
verify they are appropriate for the targeted learning outcomes.
As a result of engaging the activities created to cultivate learning outcomes, students generate
artifacts or work. The artifacts provide data useful for assessing proficiency at the student level,
the activity level, and the program level.
2.4. Accumulating evidence of student ability
Data that provides direct evidence of student ability is fundamental to the proposed model.
Program accrediting agencies, such as ABET, expect direct assessment of student work. Evidence
of student EM abilities is termed an artifact. An artifact reflects the work of an individual student
that is relevant to one or more rows of the EM learning-outcome rubrics. An artifact may take
myriad forms: a reflection paper written by a student; a video of a student presentation; a
student’s response to an exam question; or a term paper or project report and more.
A system or systems for accumulating and storing artifacts is needed. Although the details of
artifact archive systems are beyond the scope of this paper, we outline some basic parameters.
Procedures for collecting EM artifacts also need to be established. Many artifacts will be
generated in response to a class assignment. These artifacts can be collected using normal class
procedures, which may range from students handing a tangible assignment to the instructor to
uploading the assignment to a class management system (CMS) such as BlackboardTM. The
system used to collect and archive student artifacts should be compatible with the assessment
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 68
system (see below). As with all educational materials, the procedures developed should be
compliant with the Federal Educational Records Privacy Act (FERPA). The artifact collection
system should be designed such that artifacts can be retrieved when needed by those responsible
for EM assessment.
2.5. Assessment
Assessment is about gaining insights into whether or not program learning outcomes are being
realized. Assessment can be applied to the totality of the EM program or to each activity that
cultivates EM abilities. Assessment, in the proposed model, involves applying the rubrics to the
student artifacts. This may involve applying specific rubric rows, rather than a rubric in its
entirety. The rubric rows applied will be the those designated when constructing the activity that
generated the student artifact.
Artifact scoring involves applying the appropriate rubric row(s) to an artifact. For each applicable
rubric row, the artifact is scored into one of the performance levels. For the entrepreneurial
mindset rubrics, that would entail categorizing an artifact as ―does not meet expectations,‖
―developing,‖ ―meets expectations,‖ or ―proficient.‖ In Blackboard and in other LMS systems,
rubrics can be enabled via grading forms to streamline the scoring process. A grading form
(rubric) can be attached to an assignment. LMS based approaches such as this one automatically
capture how an artifact is scored on each rubric row and aggregate the scoring data. EpsilenTM
(www.epsilen.com) is the only CMS of which we are aware that enables sharing of rubrics across
courses, programs, and institutions. Rubric sharing makes it possible for one source to publish
rubrics that can then be easily used by many scorers at different institutions, as well as those
teaching different courses. The outcome of the artifact scoring process is a matrix of students by
rubric rows.
Assessment of an activity or a program is based on aggregated data. Data aggregation involves
determining the frequency for each rubric row at which each proficiency column was evident in
the student artifacts. These summaries provide a useful snapshot for answering the question,
―how are we doing?‖ Interpretation is simplified if the scores are normalized as percentages and
displayed visually as bar charts. A preponderance of scores in the ―meets expectations‖ and
―proficient‖ categories for a rubric row would suggest students are performing well at that ability.
A preponderance of scores in the ―does not meet expectations‖ or ―developing‖ categories signals
ability on which more development is needed. Low ability is expected, for example, of freshman
students making their first attempts at creating a customer appropriate value offering. A pattern of
low ability evident in the work of senior standing students, on the other hand, could point to a gap
in program design or a weakness in relevant program activities.
Figure 2 illustrates three scenarios based on simulated data for n=20 artifacts. Each bar represents
the number of student artifacts scored at each of the four proficiency levels. Let us assume that
the goal is for at least 85% of artifacts to be scored as ―meets expectations‖ or ―proficient.‖ In the
left-most scenario, 15 of the 20 artifacts were scored ―does not meet expectations.‖ Only 15% of
the artifacts reach the goal levels. This distribution reflects a lack of proficiency. In the middle
scenario, the scores are concentrated in the ―developing‖ and ―meets expectations‖ categories.
The middle scenario provides evidence that some progress is being made to achieve the learning
outcome, however, only 45% of the artifacts meet the minimum expectations. Clearly there is
room for improvement. The bar set on the right reveals that the 60% of artifacts are scored at
―meets expectations‖, and 25% of the artifacts are scored as ―proficient.‖ Thus, the goal is
realized as 85% of the artifacts are scored as ―meets expectations‖ or ―proficient.‖
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 69
Figure 2. Three assessment scenarios
Who scores the artifacts? Artifact scoring should be objective, repeatable, and performed by
knowledgeable individuals familiar with the artifacts and the rubric(s) they are applying. Units
vary in terms of how artifact scoring is handled. Faculty in the College of Engineering at ONU,
for example, are responsible for scoring artifacts generated by assignments in classes that they
teach. In contrast, ONU’s new outcome-driven general education program places the
responsibility for scoring artifacts under the jurisdiction of the University Assessment Committee.
Companies such as EduMetry provide an artifact scoring services. Departments and colleges will
need to develop artifact-scoring procedures that fit with the resources available.
A database of artifact scores should be designed to allow assessment of student ability at key
program points (e.g., freshman vs. seniors). The ability to assess intra-student development over
time is also a desirable feature, as it would allow tracking student proficiency changes over time.
Artifact score data can also be compared across institutions.
An example of how assessment data might be aggregated is provided in Figure 3. Figure 4, which
presents simulated data, illustrates how assessment data might be reported for each learning
outcome. Each bar represents the percentage of artifacts that meet or exceed the identified
minimum expectation. Reprising the standard used earlier of at least 85% of artifacts scoring at
the ―meets expectations‖ level or better, the example (Figure 4) suggests that program targeted
proficiency is achieved on all but the first two learning outcomes. This suggests that changes may
be needed in regards to how student collaboration and critical thinking abilities are cultivated.
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 70
EM Learning-outcome Does Not Meet
Expectations Developing
Meets
Expectations Proficient
Effectively collaborate in a
team setting
Apply critical and creative
thinking to ambiguous
problems
Construct and effectively
communicate a customer-
appropriate value
proposition
Persist through and learn
from failure.
Effectively manage
projects through
appropriate
commercialization or final
delivery process.
Demonstrate voluntary
social responsibility.
Relate personal liberties
and free enterprise to
entrepreneurship.
Figure 3. Example EM Program Dash Board
Figure 4. Example Program Summary Graphic
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 71
2.6. What should be changed?
Assessment enables continuous improvement. When there is evidence that student performance is
not achieving or exceeding target levels, those areas of underperformance should be examined
and adjusted. Our hypothetical score card data revealed failure to achieve target performance
levels for the effective collaboration and critical thinking learning outcomes. These failures may
be due to the activities developed to cultivate the learning outcome, problems with the rubric, or
specific rows of the rubric, or with the learning outcome. Each is discussed.
2.6.1. Reviewing then modifying existing or create new activities
Lack of performance may reflect problems with the activities designed to cultivate the learning
outcome. Examine the portfolio of activities developed to cultivate the learning outcome. The
Program Matrix, described above, will be very useful for this task. Perhaps, examination of the
Program Matrix reveals an absence of activities designed to cultivate the learning outcome: no
activities equals no development of relevant student abilities. Activity development is the obvious
next step. If the performance matrix reveals activities intended to cultivate abilities relevant to the
learning outcome, then drill down in the assessment data to determine whether failure to achieve
the target performance level can be isolated to specific rows of the rubric. Determine whether
activities exist to cultivate the identified problem rows. If no activities exist, they should be
created or existing activities should be modified to include them. If all rubric rows are addressed
by existing activities, those activities should be reviewed and modified or replaced. Activities
should be reviewed to ensure they are aligned with the abilities emphasized by the rubric.
Activities not aligned with the rubric should be modified to bring them into alignment. It is
possible that each activity will be modified in some way in response to its application and
assessment.
2.6.2. Modify the rubric?
It is possible that failure to achieve target performance levels is due to errors, omissions, or
misspecifications in the rubric. It is possible that the level of performance formalized into the
rubric is too ambitious for an undergraduate program, or not possible given available resources,
for example. It is also possible that individuals applying the rubric are misinterpreting the
language describing the performance levels. This may be discovered through conversation with
individuals that score the artifacts. These exemplify some of the reasons that assessment data may
suggest modifying a rubric to remove these sources of error or bias. After making modifications
to a rubric, the activities relevant to that rubric should be reviewed and modified, if necessary.
The assessment process should enable and inform continuous refinement of the rubrics.
2.6.3. Modify the learning-outcome?
Modifying the learning outcome is a third area of possible change. Failure to achieve
performance levels on a specific learning outcome may reflect a learning outcome that is too
broadly defined or unattainable. The assessment data patterns may suggest ways to modify the
learning outcome to one that is well defined or attainable. Changes to learning outcomes may also
be changed to reflect shifting priorities or new philosophies. Of the three areas of possible
change, modifications to learning outcomes are the least frequent.
3. Conclusions and Future Work
This article describes a learning-outcome driven approach to developing and assessing programs
to cultivate the entrepreneurial mindset in undergraduate students. The entrepreneurial mindset is
characterized by seven learning outcomes. Each learning outcome is operationalized via a rubric.
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 72
The rubrics afford specific guidance to faculty, program coordinators and others in their efforts to
create educational activities intended to develop specific abilities reflective of the entrepreneurial
mindset. The rubrics also provide a basis for assessing student performance so as to ascertain the
effectiveness of an individual activity, a course, or a program. The rubrics as presented represent
an initial effort to operationalize the seven KEEN learning outcomes.
Future work is needed to refine and validate the rubrics. This requires that faculty employ the
rubrics and provide feedback on their experiences. We emphasize that the rubrics as presented are
intended as a starting point. It is anticipated that the rubrics will evolve and improve through use
across KEEN. Further, it is possible that individual institutions may see fit to modify one or more
of the rubrics to better reflect local approaches to cultivating aspects of the entrepreneurial
mindset. At the time of this writing, rubrics for three of the seven EM learning outcomes are
undergoing field trial by faculty at participating KEEN institutions. Feedback received through
this process will be used to revise the rubrics. It is expected that two rubrics will be subjected to
field trial each academic term on an ongoing rotation.
Faculty members around the world are creating activities that cultivate some aspect of the
entrepreneurial mindset. It would seem desirable to create an easily accessible EM Activity
Repository of these activities. The repository should designate each activity according to the
entrepreneurial mindset learning outcome(s) it is designed to cultivate and also allow designation
of the rubrics and the rows of the rubrics that should be used for assessing student work generated
by the activity. A tool such as this would allow faculty or program designers, perhaps after
reviewing their Program Matrix, to quickly identify activities that cultivate specific student
abilities.
Faculty members seek EM cultivating activities that are effective. It would be desirable for the
EM Activity Repository to also accumulate assessment results from faculty who have used the
activities. It would also seem desirable that the Repository would allow faculty to submit updated
versions of an activity that reflects modifications based on experience applying the activity.
An assessment dashboard would be desirable. The dashboard would display a metric that reveals
the degree to which students in a program are realizing each of the EM learning outcomes. To
gain understanding of the effectiveness of a multi-school initiative, dashboard data could be
aggregated across institutions. Ideally, the dashboard would provide a summary at the learning-
outcome level and also allow fine-grained analysis of program effectiveness for each learning-
outcome sub-area. A dashboard that allows drilling down to the rubric-row level would yield such
fine-grained insights.
References
Association of American Colleges and Universities. (2011). Valid assessment of learning in
undergraduate education. http://www.aacu.org/value/rubrics/ (last accessed February 2011).
Arum, R. and Roska J. (2011). Academically Adrift: Limited Learning on College Campuses.
Chicago: University of Chicago Press.
Bhide, A.V. (2000). The Origin and Evolution of New Business. New York: Oxford.
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. (1956). Taxonomy
of Educational Objectives: The Classification of Educational Goals. Handbook I: Cognitive
Domain. New York: Longmans, Green, 1956.
Carlson, C.R. and Wilmot W. (2006). Innovation: The Five Disciplines for Creating What
Customers Want. New York: Crown.
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 73
Christensen, C.M., Horn, M.B., and Johnson, C.W. (2008). Disrupting Class: How Disruptive
Innovation Will Change the Way the World Learns. New York: McGraw-Hill.
Cooper, R.G. (2011). Winning at New Products: Creating Value Through Innovation. New York:
Basic Books.
Dweck, C.S. (2000). Self-Theories: Their Role in Motivation, Personality, and Development.
Florence, KY: Psychology Press.
Fink, L.D. (2003). Creating Significant Learning Experiences: An Integrated Approach to
Designing College Courses. San Francisco, CA: Jossey-Bass.
Kriewall, T.J., and Mekemson, K. (2010). Instilling the entrepreneurial mindset into engineering
undergraduates, The Journal of Entrepreneurial Engineering, 1(1): 5-19.
Ohio Northern University. (2009). Final Report of the General Education Committee.
http://www.onu.edu/files/academic_affairs_general_education_final_report_i_23april2009_a
nd_rubrics_-_updated_03252011.pdf (last accessed August 2011).
Otter, S. (1995). Learning-outcomes in higher education. In J. Burke, ed., Outcomes, Learning,
and the Curriculum: Implications for NVQ’s, GNVQ’s and Other Qualifications. Bristol, PA:
Falmer Press, 271-283.
Maxwell, (2000). Failing forward: Turning mistakes into stepping stones for success. Nashville,
TN: Thomas Nelson, Inc.
Riffe, W.J., Tavakoli, M., and Harris, M. (2010). ―Design, implementation and assessment of
entrepreneurial workshops for an entire university faculty.‖ NCIIA Open 2010,
http://nciia.org/sites/default/files/conf2010papers/riffe.pdf (Last accessed August, 2011).
Schramm, C.J. (2006). The Entrepreneurial Imperative: How America’s Economic Miracle will
Reshape the World (and Change Your Life). New York: Harper Collins.
Smith, R.M. (2008). Conquering the Content: A step-by-Step Guide to Online Course Design.
New York: Jossey-Bass.
The TLT Group: Teaching, Learning, and Technology. (2011). ―Rubrics: Definition, Tools,
Examples, References,‖ http://www.tltgroup.org/resources/flashlight/rubrics.htm (Last
accessed February 2011).
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 74
Appendix
Draft Rubrics Developed to Operationalize the Entrepreneurial Mindset
KEEN Learning Outcome: Effectively collaborate in a team setting
Student is able to contribute to team meetings in ways that advances the group’s work,
facilitates the contributions of other team members, contributes to the project effort
outside of team meetings, fosters a constructive team climate and responds effectively
to conflict that may arise within the team effort.
Does Not Meet Expectations
Developing Meets
Expectations Proficient
Con
trib
ute
s to
Tea
m M
eeti
ng
s
Does not share
ideas or solutions
that advance the
work of the
group.
Offers new ideas or solutions to advance the work of the
group.
Offers alternative ideas or solutions
or courses of action that build on the
ideas of others.
Helps the team
move forward by
articulating the
merits of
alternative ideas
or solutions.
Fa
cili
tate
s th
e C
on
trib
uti
on
s of
Tea
m M
ember
s
Does not engage
team members
by taking turns
and listening to
others without
interrupting.
Engages team members by taking turns and listening to
others without interrupting.
Engages team members in ways that facilitate their
contributions to meetings by restating the views of other
team members and/or asking questions for clarification.
Engages team members in ways that
facilitate their contributions to
meetings by constructively building
upon or synthesizing the
contributions of others.
Notices when
someone is not
participating and
invites them to
engage.
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 75
KEEN Learning Outcome: Effectively collaborate in a team setting (Continued)
Ind
ivid
ual
Co
ntr
ibu
tio
ns
Ou
tsid
e
of
Tea
m M
eeti
ngs
Does not
complete all
assigned tasks by
deadline.
Completes all assigned tasks by deadline
Work accomplished advances the project.
Work accomplished is thorough and
comprehensive.
Proactively helps
other team
members
complete their
assigned tasks to
a similar level of
excellence.
Fost
ers
Con
stru
ctiv
e T
eam
Cli
mate
Does not support
a constructive
team climate;
does none of the
following:
Supports a
constructive
team climate by
doing any two of
the following:
Supports a
constructive
team climate by
doing any three
of the following:
Supports a
constructive
team climate by
doing all of the
following:
Treats team members respectfully by being polite and constructive in
communication.
Uses positive vocal or written tone, facial expressions, and/or body language
to convey a positive attitude about the team and its work.
Motivates teammates by expressing confidence about the importance of the
task and the team's ability to accomplish it.
Provides assistance and/or encouragement to team members.
Res
po
nd
s to
Co
nfl
ict
Passively accepts
alternate
viewpoints/ideas/
opinions; fosters
conflict.
Identifies and acknowledges conflict.
Redirects focus toward common ground, toward task at
hand (away from conflict).
Stays engaged with conflict until it is
resolved.
Directly and
constructively
helps to
manage/resolve
conflict in a way
that strengthens
overall team
cohesiveness and
future
effectiveness.
Derived from the AAC&U Teamwork Value rubric
(http://www.aacu.org/value/rubrics/Teamwork.cfm)
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 76
KEEN Learning Outcome: Apply critical and creative thinking to ambiguous problems
Students are able to correctly identify the underlying problems or issues in both
theoretical and practical realms, apply appropriate analytical and creative skills to
develop feasible alternatives while considering multiple perspectives, and provide
creative and logical situations.
Section 1 of 3: Problem Identification
Does Not Meet Expectations
Developing Meets
Expectations Proficient
Iden
tifi
es k
ey
issu
e(s)
an
d
con
vert
s to
a p
roble
m s
tate
men
t
Does not show
comprehension
of the situation
to be able to
identify the key
issue(s)
Shows
comprehension
of the situation
and correctly
identifies some
key issue(s) but
does not
articulate a
reasonable
problem
statement
Shows good
comprehension
of the situation
and correctly
identifies most
key issue(s) and
articulates
reasonable
problem
statement(s)
Comprehends
situation to be
able to identify
key issue(s) and
articulates valid
problem
statement(s)
Co
nsi
der
s th
e si
tua
tio
n f
rom
mu
ltip
le p
ersp
ecti
ves
Does not
consider the
problem(s) from
different
perspectives to
view it
comprehensively
Brings out some
of the
perspectives
applicable to the
issue(s), but
leaves out some
aspects; hence,
problem
identification
suffers from this
bias/skew
Considers the
issue(s) using a
few applicable
alternate
perspectives, but
does not use this
to ―frame‖ the
key issue(s)
realistically/
originally
Considers the
issue(s) from
multiple
perspectives and
uses this to
―frame‖ the key
issue(s)
realistically/
originally
Doesn't show the
ability to
combine these
different
perspectives to
view the issue(s)
more
comprehensively
Shows evidence
of combining
some of the
alternate
perspectives to
present well
rounded issue(s)
Is able to
combine
different
perspectives and
relate them to
each other
effectively to
comprehend the
issue(s) more
comprehensively
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 77
KEEN Learning Outcome: Apply critical and creative thinking to ambiguous problems (Continued)
Section 2 of 3: Research and Analysis
Does Not Meet Expectations
Developing Meets
Expectations Proficient
Dem
on
stra
tes
ab
ilit
y to
id
enti
fy a
nd
eva
luate
in
form
ati
on
No evidence of
search, selection
or source
evaluation skills
Demonstrates
ability to search
and select
information
sources, but
evaluation of
sources is not
quite adequate
Demonstrates
adequate skills in
searching,
selecting and
evaluating
information
sources to meet
information
needs
Evidence of
search, selection
and source
evaluation skills
to meet
information
needs
Displays ability
to identify
uniquely salient
sources
Sta
tes
pla
usi
ble
rea
son
ing f
or
posi
tion
or
act
ion
s
des
crib
ed
Selects and
applies data to
situation or
rejects it without
any justification
Applies some
rudimentary
reasoning in
consideration
and application
of data to
situations, which
is often
incorrect/
inadequate
Spells out the
reasoning
applied in
consideration
and application
of data to
situations with
few lapses in
adequacy of data
Spells out the
reasoning
applied in
consideration
and application
of data to
situations
Analysis is
inappropriate,
inadequate and
superficial, at
best
Uses some
analytical tools/
methods
Analytical tools
and methods
employed are
appropriate and
about accurate,
but not the most
suitable ones,
against the need
Analytical tools
and methods
employed are
appropriate and
adequate for the
need
An
aly
zes
info
rmati
on
usi
ng
appro
pri
ate
co
nce
ptu
al
fram
ewo
rk/t
oo
ls
Does not explain
which specific
tools or methods
are relevant to
the issue(s) at
hand
Analysis is
minimal against
the need and
does not fully
clarify the issues
and facilitate
decision-making
Analysis
somewhat
clarifies the
issues and
facilitates
decision-making
Analysis clarifies
the issues and
facilitates
decision-making
Does not clearly
and explain
which specific
tools or methods
are relevant to
the issue(s) at
hand
Partly explains
why specific
tools or methods
are relevant to
the issue(s) at
hand
Explains why
specific tools or
methods are
relevant to the
issue(s) at hand
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 78
KEEN Learning Outcome: Apply critical and creative thinking to ambiguous problems (Continued)
Section 3 of 3: Solution Development
Does Not Meet
Expectations Developing
Meets
Expectations Proficient
Dis
pla
ys a
bil
ity
to e
xplo
re a
lter
nate
opti
on
s to
th
e id
enti
fied
pro
ble
m
Does not come
up with alternate
feasible and
original options
Makes an effort
to minimally
explore alternate
solutions but
plays safe, does
not risk
uncertainty
associated with
original solutions
Explores
alternate
solutions to the
problem but
these are not
very original,
tends to play safe
to an extent
Explores and
comes up with
original, relevant
alternate
solutions to the
problem
Ma
kes
con
vin
cin
g a
rgu
men
ts f
or
reco
mm
ended
opti
on
, ali
gn
ed t
o
the
giv
en c
on
text
Fails to identify
conclusions,
implications, and
consequences of
the issue
Does not clearly
identify
conclusions,
implications, and
consequences of
the issue
Analyzes
conclusions,
implications, and
some
consequences of
the issue
Analyzes
conclusions,
implications, and
the majority of
consequences of
the issue
Doesn’t propose
solution to
problem
Proposes
solution to
problem(s)
without much
investigation and
analysis
Proposes
solution to
problem(s) with
some
investigation and
analysis
Proposes
solution to
problem(s) based
on detailed
analysis
Justifies the
recommended
option and
recognizes the
implications
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 79
KEEN Learning Outcome: Construct a customer-appropriate value proposition
This rubric is based on the NABC model for expressing value propositions as presented
in Carlson and Wilmot’s book Innovation: The Five Disciplines for Creating What
Customers Want.
Does Not Meet
Expectations Developing
Meets
Expectations Proficient
Import
an
t cu
stom
er a
nd m
ark
et
nee
d (
op
po
rtu
nit
y)
Customer and
market needs not
identified.
Prospective customer and market needs and
characteristics are identified and described.
Identified customer and market
needs and characteristics are
quantified.
All key
constituencies
are identified
(e.g., target
customer,
investors); the
above are
replicated for
each key
constituency.
Wh
at
is t
he
un
iqu
e ap
pro
ach
for
addre
ssin
g t
he
nee
d?
No approach for
addressing the
need is provided.
An approach for addressing the need is described.
The approach is appropriate relative to the identified
customer and market needs.
The approach
evidences
incremental
uniqueness
relative to
existing
alternatives.
The approach
has a
distinguishable
advance in
design,
technology,
process, or
business model.
The approach
has an important
advance in some
combination of
design,
technology,
process, or
business model.
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 80
KEEN Learning Outcome: Construct a customer-appropriate value proposition (Continued)
Does Not Meet
Expectations Developing
Meets
Expectations Proficient
Wh
at
are
th
e sp
ecif
ic B
enef
its/
Cost
s th
at
resu
lt f
rom
th
is
appro
ach
No customer
benefits per costs
are provided.
Key customer benefits of the approach are identified
using specific language.
Customer costs of the approach are
identified using specific language.
The identified
key customer
benefits are
quantified in an
appropriate way.
The identified
customer costs of
the approach are
quantified in an
appropriate way.
Benefits relative
to costs of the
approach are
clearly
expressed.
How
are
th
ese
ben
efit
s/co
sts
super
ior
to a
vail
able
alt
ern
ati
ve s
olu
tio
ns
for
solv
ing t
he
pro
ble
m?
No alternative
solutions are
mentioned.
Key direct alternative solutions are identified and the
benefits and costs of each are quantified and explained in
specific language.
Key indirect alternative solutions are
identified and the benefits and costs
of each are quantified and explained
in specific language.
The superiority
of the proposed
alternative’s
benefits and
costs relative to
key direct and
indirect
alternatives is
quantified and
explained in
specific
language.
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 81
KEEN Learning Outcome: Construct a customer-appropriate value proposition (Continued)
Does Not Meet
Expectations Developing
Meets
Expectations Proficient
How
are
th
e ben
efit
s/co
sts
of
the
pro
po
sed s
olu
tio
n
super
ior
to t
he
Com
pet
itio
n?
No competition
is mentioned.
Key direct competitors are identified and the benefits and
costs of each are quantified and explained in specific
language.
Key indirect competitors are
identified and the benefits and costs
of each are quantified and explained
in specific language.
The superiority
of the proposed
alternative’s
benefits and
costs relative to
key direct and
indirect
competitors is
quantified and
explained in
specific
language.
Mu
ltip
le v
alu
e
pro
posi
tion
s fo
r m
ult
iple
con
stit
uen
cies
No
constituencies
identified.
All key constituencies are identified.
An effective
value proposition
is developed for
each of some of
the identified key
constituencies.
An effective
value proposition
is developed for
each key
constituency.
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 82
ONU General Education Learning Outcome: Effective communication of a customer appropriate value proposition
Students demonstrate the written, oral and visual communication skills necessary to
communicate professionally and effectively as responsible members of their
organizations and their communities.
Section 1 of 3: Written Communication
Does Not Meet
Expectations Developing
Meets
Expectations Proficient
Ove
rall
Org
an
iza
tio
n Thesis statement
is unclear or
absent
Provides a thesis
statement that is
somewhat
developed
Provides a thesis
statement that is
generally clear
Provides a
strong, clear
thesis statement
Provides a
structure and
organization that
is not cohesive
or coherent
Provides a
structure and
organization that
is somewhat
cohesive and
coherent
Provides a
structure and
organization that
is generally
cohesive and
coherent
Provides a
structure and
organization that
is strongly
cohesive and
coherent
Para
gra
ph
Dev
elopm
ent
Writes
paragraphs that
are
underdeveloped,
with topic
sentences that
are missing or
unsupported
Writes
paragraphs that
are developed
inconsistently,
with topic
sentences that
are present, but
not fully
supported
Writes
paragraphs that
are generally
well-developed,
with topic
sentences that
are present and
supported
Writes
paragraphs that
are well-
developed, with
strong, focused
topic sentences
that are fully
supported
Form
at
an
d S
tyle
Demonstrates
improper use of
appropriate
document
formatting
guidelines.
Demonstrates
inconsistent use
of appropriate
document
formatting
guidelines.
Demonstrates
appropriate use
of appropriate
document
formatting
guidelines.
Demonstrates
excellent use of
appropriate
document
formatting
guidelines.
Integrates
quotations and
exhibits
inappropriately
Integrates
quotations and
exhibits
inconsistently
Integrates
quotations and
exhibits
appropriately
Integrates
quotations and
exhibits
effectively into
the analysis
Wri
tten
La
ng
ua
ge
Mech
an
ics
Writes sentences
that are unclear
or indirect
Writes sentences
that are
occasionally
unclear, indirect
or grammatically
incorrect
Writes sentences
that are generally
clear, concise,
and direct.
Writes sentences
that are
consistently
clear, concise,
and direct and
grammatically
correct
Minor problems
with grammar
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 83
ONU General Education Learning Outcome: Effective communication of a customer appropriate value proposition
(Continued) Section 2 of 3: Oral Communication
Does Not Meet
Expectations Developing
Meets
Expectations Proficient
Ove
rall
Org
an
izati
on
of
pre
sen
tati
on
The central idea
is not clearly
stated and
remains unclear
or unidentified.
The central idea
is implied but not
explicitly stated
or communicated.
The central idea
of the
presentation is
completely and
clearly
presented.
Proposes a clear
central idea.
The presentation
moves from one
point to the other
in a disconnected
manner, without
regard to the
objective.
The
organizational
pattern of the
presentation does
not adequately
meet the
audience’s needs
or develop the
thesis
appropriately.
The
organization of
the presentation
is generally
logical and
meets the
audience’s
needs.
The entire presentation
is a cohesive piece in
which the ideas
developed support the
thesis.
Key ideas are
easily overlooked
and important
aspects of the
thesis are not
addressed.
There is lack of
clarity regarding
the purpose and
only a few of the
key ideas have
been addressed.
Presentation
begins with a
clear sense of
purpose, but the
presence of
irrelevant
content in some
parts leads to
digression from
the central idea.
The presentation leads
naturally to a strong
conclusion that
summarizes the key
points and leaves the
audience with a clear
message.
Arg
um
ent
an
d R
het
ori
c
Presentation lacks
an identifiable
central message.
Presentation
contains an
identifiable
central message,
but it argues from
a false premise or
attempts to prove
the null
hypothesis.
Presentation
contains a
significant
central message,
with valid and
true outcome
adequately
supported by
evidence and
logic.
Presentation contains a
compelling central
message, with a valid
and true outcome fully
supported by evidence
and logic.
Presentation does
not rely on
credible evidence,
or form a valid
and true outcome.
Argument is not
fully supported by
credible evidence;
the outcome is
either invalid or
valid but untrue.
Able to answer
basic objections
to their
conclusions.
Able to answer basic
and (at least some)
more
advanced/sophisticated
objections to their
conclusions.
Unable to answer
basic objections to their
conclusions
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 84
ONU General Education Learning Outcome: Effective communication of a customer appropriate value proposition
(Continued) Section 2 of 3: Oral Communication
Does Not Meet
Expectations Developing
Meets
Expectations Proficient
Ove
rall
Org
an
izati
on
of
pre
sen
tati
on
The central idea
is not clearly
stated and
remains unclear
or unidentified.
The central idea
is implied but not
explicitly stated
or communicated.
The central idea
of the
presentation is
completely and
clearly
presented.
Proposes a clear
central idea.
The presentation
moves from one
point to the other
in a disconnected
manner, without
regard to the
objective.
The
organizational
pattern of the
presentation does
not adequately
meet the
audience’s needs
or develop the
thesis
appropriately.
The
organization of
the presentation
is generally
logical and
meets the
audience’s
needs.
The entire presentation
is a cohesive piece in
which the ideas
developed support the
thesis.
Key ideas are
easily overlooked
and important
aspects of the
thesis are not
addressed.
There is lack of
clarity regarding
the purpose and
only a few of the
key ideas have
been addressed.
Presentation
begins with a
clear sense of
purpose, but the
presence of
irrelevant
content in some
parts leads to
digression from
the central idea.
The presentation leads
naturally to a strong
conclusion that
summarizes the key
points and leaves the
audience with a clear
message.
Arg
um
ent
an
d R
het
ori
c
Presentation lacks
an identifiable
central message.
Presentation
contains an
identifiable
central message,
but it argues from
a false premise or
attempts to prove
the null
hypothesis.
Presentation
contains a
significant
central message,
with valid and
true outcome
adequately
supported by
evidence and
logic.
Presentation contains a
compelling central
message, with a valid
and true outcome fully
supported by evidence
and logic.
Presentation does
not rely on
credible evidence,
or form a valid
and true outcome.
Argument is not
fully supported by
credible evidence;
the outcome is
either invalid or
valid but untrue.
Able to answer
basic objections
to their
conclusions.
Able to answer basic
and (at least some)
more
advanced/sophisticated
objections to their
conclusions.
Unable to answer
basic objections
to their
conclusions
Operationalizing and Assessing the Entrepreneurial Mindset:
A Rubric Based Approach
Volume 2, Number 1 Winter 2011 85
KEEN Learning Outcome: Demonstrate voluntary social responsibility
Students display an understanding of personal, civic and social needs; identify
underlying ethical issues in specific situations; and make informed, ethical responses to
those needs.
Does Not Meet
Expectations Developing
Meets
Expectations Proficient
Un
der
sta
nd
ing
of
per
son
al,
civi
c, a
nd s
oci
al
nee
ds Does not base
understanding of
the needs of
persons, civil
societies, or the
global
community on a
body of pertinent
information.
Bases
understanding of
the needs of
persons, civil
societies, or the
global
community on a
limited body of
pertinent
information.
Bases
understanding of
the needs of
persons, civil
societies, or the
global
community on a
substantive body
of pertinent
information.
Bases
understanding of
the needs of
persons, civil
societies, or the
global
community on a
substantive and
diverse body of
pertinent
information.
Iden
tify
Un
der
lyin
g
Eth
ical
Issu
es i
n
Spec
ific
Sit
uati
on
s
Does not identify
ethical issues in
personal, civic,
or global life.
Identifies ethical
issues in
personal, civic or
global life.
Identifies ethical
issues;
recognizes that
these ethical
issues shape
personal, civic,
or global life.
Discerns the
underlying
needs, values,
and perspectives
that create
ethical issues in
personal, civic,
or global life.
Ma
ke I
nfo
rmed
, E
thic
al
Res
pon
ses
to
Th
ose
Nee
ds
Responds to
ethical issues
without regard to
pertinent
information;
does not apply
an analytical
framework.
Responds to
ethical issues by
considering
pertinent
information;
does not defend
chosen position
within an
applicable
analytical
framework.
Responds to
ethical issues by
considering
pertinent
information;
defends chosen
position within
an applicable
analytical
framework.
Responds to
ethical issues by
considering
pertinent
information;
defends chosen
position within
an applicable
analytical
framework;
evaluates
assumptions and
implications of
divergent ethical
perspectives.
R. E. Kleine, III * and J.D. Yoder
The Journal of Engineering Entrepreneurship 86
KEEN Learning Outcome: Relate personal liberties and free enterprise to entrepreneurship.
The student is familiar with defining attributes of entrepreneurial behavior, can explain
the relationship between personal liberties and entrepreneurship, and can connect the
concept of free enterprise to entrepreneurial behavior.
Does Not Meet
Expectations Developing
Meets
Expectations Proficient
En
trep
ren
eurs
hip
No defining
attributes of
entrepreneurial
behavior are
identified.
Key defining attributes of entrepreneurial behavior are
identified.
Can identify attributes of successful
entrepreneurial behavior.
Provides relevant
examples to
successful
entrepreneurial
behavior.
Rel
ate
per
son
al
liber
ties
to
entr
epre
neu
rsh
ip
No evidence of
relating personal
liberties and
entrepreneurial
behavior.
Some evidence
of relating
personal liberties
and
entrepreneurial
behavior.
Can describe the relationships
between personal liberties and
entrepreneurial behavior.
Provides relevant
examples to
illustrate the
relationships
among personal
liberties and
entrepreneurial
behavior.
Rel
ate
fre
e en
terp
rise
to
entr
epre
neu
rsh
ip
No evidence of
relating free
enterprise and
entrepreneurial
behavior.
Some evidence
of relating free
enterprise and
entrepreneurial
behavior.
Can describe the relationships
between free enterprise and
entrepreneurial behavior.
Provides relevant
examples to
illustrate the
relationship
between free
enterprise and
entrepreneurial
behavior.