detwannabe.files.wordpress.com€¦ · web viewit was the researchers’ intent to investigate...

Running head: THIRD CRITICAL REVIEW

Third Critical Review of Research

Personalized workplace learning: An exploratory study on digital badging within a teacher

professional development program

Jeremy T. Bond

EDU 800 - Central Michigan University

Dr. Michael DeSchryver

December 6, 2015

THIRD CRITICAL REVIEW2

1. Identify the clarity with which this article states a specific problem to be explored.

When review is confined to the earliest portion of Gamrat, Zimmerman, Dudek and

Peck’s (2014) article a clearly stated problem is found not long into the article. However, the

authors first offer a brief working definition for and overview of digital badging, touting

advantages and describing general characteristics such as the “valued encapsulation of

experiences…stored metadata, such as the issuer, description and evaluation criteria” (2014, p.

1136). Badges, or rather collections of badges are also described as being able to “act like a

portfolio, documenting workplace learning… that can be shared” (2014, p. 1137). Shortly

hereafter the problem to be explored is indicated as the authors share that the “team is exploring

digital badges as tools that support professionals’ decision making within their own PD

activities” and “analyze… implementation of an online teacher PD system…” (p. 1137). The

technique of offering foundational detail is appropriate considering the authors’

acknowledgement that digital badging is relatively new. Offering the uninitiated reader some

context related to the research is valuable; as it helps novice writers negotiate meaning (Swales,

1990).

2. Comment on the need for this study and its educational significance as it relates to this problem.

A lack of research appears to be the primary motivator; however a gap in existing

scholarship alone is not a reason for new research. The need for this research is better supported

by accepted beliefs related to teacher professional development including a desire to “tell a story

about the learner’s education and skills” (Gamrat, Zimmerman, Dudek, & Peck, 2014, p. 1136)

as well as growth in need, increased workplace diversity, and individualization. Moreover,

“although the educational sector has realized the potential for investing in online teacher PD, a

2


one-size-fits-all approach…cannot give ongoing support” (p. 1138). As PD options become

greater, more customized, and available on demand online, teachers will experience greater

choice and more flexibility, while simultaneously having a greater need to track and maintain

records of completion and meaningful information related to these records. With the

aforementioned tenets in mind, research involving digital badges, personalization, customization

and choice is a logical endeavor.

In addition, earlier studies considered by the authors also call for work in their chosen

area. In the conclusion of their work, McLoughlin and Lee (2008) indicate “technological

resources provide opportunities for a range of interactions, communicative exchanges, and

sharing, but it is not possible to base an entire sequence of learning episodes solely on tools.”

Though digital badges did not arrive on the scene until 2010 (Ash, 2012), McLoughlin and Lee

seem to make a forward-looking allusion to the great potential for such tools as contributors to

interaction, choice, etc. Likewise, further support of this work is indicated by Stoll, Bolam,

McMahon, Wallace, and Thomas’s (2006) claim, as cited in Gamrat, Zimmerman, Dudek, &

Peck (2014), “professional learning is widely believed to be more effective when it is based on

self-development and work-based learning” (p. 232).

3. Comment on whether the problem is “researchable”? That is, can it be investigated through the collection and analysis of data?

The problem explored is indeed “researchable.” It was the researchers’ intent to

investigate digital badges as a tool to support teacher PD decision making as well as badging’s

potential to “support the needs of personalized professional learning” in a “technologically

enhanced workplace” – in this case, the workplace being American primary and secondary

schools. For this endeavor data must be gathered relating to participant (teacher) choice,

3


interactions with the online system (for badging and training), as well as insight into the decision

making process and what impacts were had on it by the presence of customizability,

personalization and the presence of the badges. To address these needs participant activity was

examined within the construct of TLC, an “online PD website that uses digital badging to

provide and capture ‘real-time, ongoing work-embedded support” (Gamrat, Zimmerman, Dudek,

& Peck, 2014, p. 1140), which yielded two of the study’s four data sources, the only two

germane to all participants, which were “system generated records” and “PD activity logs” (p.

1141). Additional data was gathered from a smaller sample set which included purpose

statements and “pre- and post-TLJ interviews” (p. 1141).

The use of multiple data sources aids in triangulation, thereby, helping to establish

validity. Patton (2002) cautions again triangulation being used to arrive at consistency and

suggests instead it is best used to discover deeper understandings. The researchers in this case

sought to gain understandings necessary to address Borko’s (2004) critiques of PD. Through her

multi-focal lense metaphor, Borko (2004) notes the need to “focus on the individual teacher” (p.

8) from a situational perspective on PD.

4. Critique the author’s conceptual framework.

A conceptual framework is essentially the system of concepts, assumptions, expectations,

beliefs, and theories that supports and informs your research, which is to say the things to be

studied…and relationships among them (Maxwell, 2005). As stated, the framework in Gamrat,

Zimmerman, Dudek, & Peck (2013) was “adapted…from the educational technology

personalization literature provided by Kearney, Schuck, Burden and Aubusson” (Gamrat,

Zimmerman, Dudek, & Peck, 2013, p. 1139). The indicated elements consist of decision making

4


and customization, respectively, for personal interests and workplace opportunities. Decision

making is defined by the researchers as “learning related to personal self-development”, while

customization is explained as “addressing learning related to workplace affordances and

constraints” (p. 1139).

The authors’ characterization of the study’s theoretical framework, though clear and

detailed in what it offers, is incomplete. Gamrat, Zimmerman, Dudek, & Peck (2013) indicate

consideration of “the role of expert feedback for the eight case study teachers as they

progressed…” (p. 1142) but do not address feedback as an element of their framework.

Furthermore, though digital badging plays prominently in the research title and stated questions,

its actual impact on the actions of the teachers, its motivational potential is also absent in the

framework considered. Despite the omission, the impact of badging seems to be assumed, as

stamps and badges were used to represent levels of learning (i.e. achievement) (p. 1142). As it is

the theoretical framework’s role in part to “[articulate] the theoretical assumptions of a research

study” and therefore “specifies which key variables influence a phenomenon of interest” (USC,

2015, n.p.), completeness matters.

5. How effectively does the author tie the study to relevant theory and prior research? Are all cited references relevant to the problem under investigation?

The literature review offers a number of focus points and connects most back to their

own research. They began by discussing factors relevant to technology integration, including

teacher attitudes, intents, and confidence as a means to establish teacher decision-making’s

importance, citing Ching and Hursh (2014). The citation notwithstanding, the connection

between teacher attitudes and outcomes related to classroom technology integration and PD-

related matters is not a direct connection. From here, the researchers indicate their work

5


addresses two earlier questions raised by Dede, Ketelhut, Whitehouse, Breit and McClosky

(2009), specifically “how should PD be designed and what types of interactions should PD

support?” (p. 1137). While research questions posed later make no overt connection to the

aforementioned inquiry, the relationship between Dede, Ketelhut, Whitehouse, Breit and

McClosky’s (2009) research agenda for PD and Gamrat, Zimmerman, Dudek, & Peck (2013) is

relevant. Finally, connections between the research and past work relating to the unique needs of

teachers, their diverse experiences and expertise, and the resulting need for personalized,

customizable experiences are made with citations to Darling-Hammond, Porter, Garet, Yoon &

Bransford (2005) and Borko (2004).

Much attention is also given to the Teacher Learning Journeys (TLJ), moreso than any

other single element. Readers learn of TLJ’s origins, contributors, and goals. It is well-

established as a platform to address “concerns that workplace learning for educators was based

on an outdated one-size-fits-all model…that does not meet the needs…” (p. 1138). The value

herein is in properly sharing the way in which the badging and training environment was selected

– not at random, and especially not with any thought that all online systems are equal in mind.

6. Does the literature review conclude with a brief summary of the literature and its implications for the problem investigated?

Though the literature review itself lacks a separate conclusive section, the final lines of

the TLJ section of the larger review serves well as a summary of the literature and implications.

It noted “the demands placed on teachers need to be balanced with their workplace…constraints”

and acknowledged “questions about PD experiences and how to track their effects over time” (p.

1138). In terms of implications, the work offered the importance of “a teacher’s ongoing needs”

and a view of how “choices and feedback…inform their own PD decisions” (p. 1138). In

6


general, the researchers succeed at identifying one-size fits all approaches to PD as unsuccessful

and out of date. Further, they support the choice of TLJ as a platform for PD and PD research.

What is less clearly established is a connection between how traditional and online digital-badge

driven PD relates to decision making or customization theory.

7. Evaluate the clarity and appropriateness of the research questions or hypotheses.

The authors of this study offered two research questions as follows, on page 1138:

- How do teachers use a digital badging system to personalize their experience in TLJ to meet their PD goals (decision making)?

- How do teachers personalize their PD activities within TLJ to support unique aspects of their workplace setting (customization)?

The parenthetical inclusions of decision making and customization appear to force

connections between these questions to the two primary elements of the theoretical framework.

In the absence of the parentheticals, however, both questions regard personalization. The chief

difference between the two questions relates to the purpose of the respective “hows” – whether

personalization is used to modify experience in pursuit of individual goals, or whether, in the

second question, the activities are personalized to accommodate workplace affordances and

constraints. As personalization is an element of both pieces of the study’s framework, and plays

heavily into a literature review positing that PD settings which preclude such freedoms are

proving ineffective in modern teaching environments, the questions’ focus is appropriate.

Clarity, however, is lacking.

According to Vanderbilt University’s Writing Center, as cited in Duke University’s

Thompson Writing Program’s guide, What Makes a Good Research Question, “…a good

research question allows the writer to take an arguable position, it DOES NOT leave room for

7


ambiguity.” (p. 2). Furthermore, the best social sciences questions seek to achieve one of four

particular outcomes, summarized as defining or measuring a specific fact/facts about a

phenomenon; match facts and theory; evaluate/compare two theories, etc.; or, prove a certain

method is more effective than other methods (Duke, n.d.).

Each question in this study, though clear, indicates it will explore the “how” of a

system’s use to particular ends. Ultimately, though the ways in which users interacted with the

system – the how of their use – is not what’s delivered. The information put forth, as will be

noted later, is much more of a “what” than a “how”.

8. Critique the appropriateness and adequacy of the study’s design in relation to the research questions or hypotheses.

As the title indicate the researchers engage in an exploratory study, not surprisingly as

exploratory research’s purpose is “about putting one’s self deliberately in a place – again and

again – where discovery is possible…” and “…interests can be pursued.” (Stebbins, 2001, n.p.).

Underlying their exploration is a collective case study, wherein the researchers investigate much

more closely the activities of eight of the thirty-six participants. This design allows, as indicated

by Baxter and Jack (2008), for “the researcher to analyze within each setting and across

settings,” which is appropriate given eight different teacher’s involvement, in different school

contexts, etc.

9. Critique the adequacy of the study’s sampling methods (e.g., choice of participants) and their implications for generalizability.

The sample size in this study is modest, but reasonable for this type of exploratory case

study research. A total of thirty-six (36) teachers served as participants, a sample size not of

concern. It is also indicated that the teachers were “at all levels of their careers” (p. 1141),

8


though precisely what is meant by this is not shared, a range of years of experience – from first

year, to well-seasoned educators were included.

A minor concern is the way in which the participants were found – “via email rosters

from partnering educational organizations” (p. 1141). This method indicated two things:

participants who do not use email were not considered and participants must be affiliated with

the selected educational organizations, which were not named. Though email use is prevalent

(85% of teachers use email according to a 2010 U.S. Dept. of Education report), its use is not

universal. Furthermore, depending on the nature of the educational organizations, teachers with

existing interests in certain types of professional development opportunities and constructs may

have been more likely to be recruited.

From among the thirty-six who registered an interest, all thirty-six were accepted.

Among these, eight (8) were selected to take part in deeper exploration in a “collective case

study method” (p. 1140). The eight case study participants were chosen based on criteria seeking

balance in years of experience. Moreover, the eight involved in the deeper study had to also

complete all interview exercises, a purpose statement and at least one PD activity (p. 1141). The

authors acknowledge two flaws in the sample – a majority of females (with only one male) and

the requirement for Internet access among them. Though a reasonable requirement, Internet

access speaks to the same potential for skewing perspectives as does the recruitment from among

teachers using email, affiliated with partner organizations mentioned earlier. That is, teachers

with certain a predisposition toward and experience with technology were involved.

The generalizability of Gamrat, Zimmerman, Dudek, & Peck’s (2013) research is limited,

however, generalizability is not the goal of exploratory study, as noted earlier. In this sense, the

9


sample is adequate to see teachers in PD environments, again and again, making choices and

discovering the interplay among elements of the framework. Even if this were not the case,

generalizability is still likely limited to future use of TLJ among educators in a select field of the

sciences at the primary and secondary levels.

10. Critique the adequacy of the study’s procedures and materials (e.g., interventions, interview protocols, data collection procedures).

A full critique of this study’s procedures and materials is difficult, as so little about the

relevant elements (e.g. interventions, interview protocols, procedures, etc.) is revealed. Much of

what can be adequately discussed is a combination of the use of TLJ and that which can be

extracted or assumed about procedures and protocols from a combination of the minimally

detailed data analysis section and findings indicated later in the study report.

The way in which TLJ was used is the most clearly disclosed piece of the study’s

procedures. TLJ “offered teachers a set of 63 PD activities…within three science content areas”

which “provided teachers a chance to customize their learning based on workplace goals” (p.

1140). Completion of the learning activities yielded one of two types of microcredentials –

stamps or badges, following activity log review by “expert practitioners.” Relative to badges,

“stamps represented a lower level of achievement for completing activities” (p. 1141).

Analytical memos were created based upon “coding each teacher’s data” in order to

“deeply understand each teacher” (p. 1141). The “memos were also used to compare across the

group for” coding purposes (p. 1141). Given the researchers’ stated goal to conduct “theory-

driven thematic analysis” (p. 1141), analytical memos are an appropriate choice, as these are

essentially written summaries about what is thought to be learned or observed (Lofland &

10


Lofland, 1995). While the study report refers to table 1 for the thematic codes gathered, coding

data actually appears in table 2 (p. 1142). Table 1, instead, offers data relating to the quantity of

microcredentials earned and activities completed across the three science areas.

11. Critique the appropriateness and quality (e.g., reliability, validity) of the measures used.

The authors indicate findings are shared in three sections, included are a description of

the 36 teachers’ decision-making and customizations related to assessment and content,

demonstration of the eight case study teachers’ customization of PD to support workplace

characteristics, and a description of the role of expert feedback (p. 1141-1142).

Quantitative data is presented relative to teacher choice of lower level (stamp) and higher

level (badge) achievements. Of all microcredentials earned, 86.4% were stamps, which the

researchers posited “that providing teachers the ability to make decisions about their assessment

provided them the flexibility to personalize their PD” (p. 1143). The validity of this conclusion

is suspect, as other reasons – including a desire to avoid the greater reflection requirements

associated with badge completion – could as easily account for the preference toward stamp-

earning activities.

Other assertions are derived from comments made in the case study interviews and the

decisions made specifically by the eight case study participants. These include anecdotal valuing

of the expert feedback and certain preferences for or interest in specific content (e.g. climate

change). Potentially more interesting, data-supported, and relevant conclusions are absent.

These may have included data points from the activity logs such as when and how often teachers

pursued PD activities during the June to August, 2012, period of observation, as well as the order

11


in which activities were pursued, all of which better speaks to the research question relating to

how teachers used the PD environment.

12. Critique the adequacy of the study’s data analyses. For example: Have important statistical assumptions been met? Are the analyses appropriate for the study’s design? Are the analyses appropriate for the data collected? –Omitted

13. Critique the author’s discussion of the methodological and/or conceptual limitations of the results.

The authors offered discussion on four relevant areas derived from their investigation.

The first three are pertinent to Borko’s (2004) assertion “that teachers can benefit from flexibility

within PD…” (p. 1145) and include a need for flexible organization, varied delivery of content

and assessment measures, and affordances relating to continuing access, such that teachers could

“archive and revisit their PD activity” (p. 1146). The fourth area of discussion regards the role

of expert mentor feedback, wherein anecdotal evidence is offered from a single case (Erin’s)

from among the eight in the collective case study.

While all of the above appear relevant to the research, the conceptual limitations are

significant, as readers hoping to apply the implications of this research would be unable to do so

responsibly. As we lack any sense of precisely what data was gathered from logs, knowledge of

the questions asked of the case study interviewees, and have nothing more than a scant number

of quotes from interviews and thematic comments from purpose statements, true implications are

difficult to identify.

One conclusion does deserve note and is supported by the limited data presented, as the

findings do suggest as indicated, “online workplace training” should provide “a wide library of…

experiences that vary in terms of content and pacing” (p. 1146).

12


14. How consistent and comprehensive are the author’s conclusions with the reported results?

As indicated earlier, the conclusion most consistent with reported results is that relating

to the potential value in an online PD environment which affords users options and allows

pursuit at one’s own pace. This conclusion supports the results, as the thirty six teachers pursued

many different activities, not one pursued all sixty-three, nor did any one of them necessarily

follow the same path as any other with respect to either which or when. Though, admittedly, this

last claim is not well-supported in the data, taken together, these elements support the potential

value of flexibility. Beyond this, however, it seems the authors were nearly desperate to offer

additional conclusions, which were simply not supported by the data offered.

15. How well did the author relate the results to the study’s theoretical base?

The two-element theoretical base offered by the authors, relating to customization and

decision making is connected within the presentation of the results. Early on, the authors told

readers that “professional development can be personalized with opportunities for decision-

making and customization” (p. 1137), and in addition, that teacher workplaces have particular

opportunities and constraints, while simultaneously requiring PD. Though, as noted in the

critique of the framework, other elements were absent, these too were related within the results,

as significant focus – if only supported by a single case – was given to the role expert mentoring

provides. Identified as early as 1959, Herzberg’s two factor theory of motivation indicated

motivation comes as a result of stimulation, accountability, and the provision of fulfillment

(Loiseau, 2011). While Herzberg did not likely envision a cloud-based PD system awarding

stamps and badges, it does seem his work was forward looking, especially in identifying factors

of an intrinsic nature, achieved from activity (Loiseau, 2011), as stamps and badges serve in this

capacity. In doing so, the microcredentials, the environment itself, and expert feedback,

13


juxtaposed by workplace constraints, all interact and impact a teacher’s use of the PD system,

decision making, and personalization of pursuits.

16. In your view, what is the significance of the study, and what are its primary implications for theory, future research, and practice?

The significance of this study is limited as in the nearly four-years since it occurred, we

know the proliferation of digital badging has remained limited, while the growth of cloud-based

(online) education systems has grown significantly. A January 2015 article still describes

badging only as “gaining traction” and indicated one can “earn one badge in a traditional

university classroom, another for participating in a MOOC, and yet another from a professional

organization for completing a training course” (Kim, 2015). Though Kim (2015) is attempting to

show flexibility of badges and how they are obtained as a positive, this same flexibility runs

counter to the desire for one to port and display their achievements and the underlying

competencies, which is something badges are also meant to do.

Setting aside the unpredictability of technological advances, and a focus on badges, there

is other significance to be found in this work. Specifically, it generally makes a good case for

teachers’ need for flexibility, discipline-specific relevance, and currency in PD activities. It

further indicates the value – if only demonstrating it in a limited fashion – of expert mentorship.

This is to say, even in the most sophisticated and well-conceived technology-rich online spaces

(such as TLJ) the feedback from and involvement of an expert, another human being, remains

special. Though I do not concur with the authors’ closing implication, that further consideration

be given to design digital-badging based systems (p. 1146), I do find significant cause to devote

further research to design of online training systems in general, be they badge-issuing or not.

14


The single greatest implication for practice, in my opinion, would be for teacher

educators to recognize the value of variety and choice, and furthermore to understand that the

activities undertaken may speak not only to a learner’s interest, but as well, the nature of the

activity, accessibility, assessment required, and the affordances and constraints of the learner’s

environment. This, in its entirety, is an important consideration for training and development

design.

15


References

Ash, K. (2012). ‘Digital Badges’ would Represent Students’ Skill Acquisition: Initiatives seek to

give students permanent online records for developing specific skills. Education Week.

Retrieved: http://www.edweek.org/dd/articles/2012/06/13/03badges.h05.html

Baxter, P. and Jack, S. (2008). Qualitative Case Study Methodology: Study Design and

Implementation for Novice Researchers. The Qualitative Report 13(4), 544-559.

Borko, H. (2004). Professional development and teacher learning: mapping the terrain.

Educational Researcher, 33(8) 3–15.

Duke University (n.d.). What Makes a Good Research Question? Writing Studio: Thompson

Writing Program.

Gamrat, C., Toomey, H., Zimmermanman, J., and Peck, K. (2014). Personalized workplace

learning: An exploratory study on digital badging within a teacher professional

development program. British Journal of Educational Technology. 45(6), 1136-1148.

Kim, J. (2015) The Value Problem in Digital Badging. Inside Higher Ed. Retrieved:

https://www.insidehighered.com/blogs/technology-and-learning/value-problem-digital-

badging

Lofland, J., and Lofland, L. (1995). Analyzing Social Settings: A Guide to Qualitative

Observation and Analysis. Wadsworth Publishing Company: Belmont, CA.

Loiseau, J. (2011, September). Herzberg’s Theory of Motivation. ACADEMIA. Retrieved:

http://www.academia.edu/901041/Herzbergs_Theory_of_Motivation

16


Maxwell, J.A. (Ed.). (2005) Qualitative Research Design: An Interactive Approach (2nd ed.).

Thousand Oaks, CA: Sage

McLoughlin, C. & Lee, M. J. (2008). The three p’s of pedagogy for the networked society:

personalization, participation, and productivity. International Journal of Teaching and

Learning in Higher Education, 20(1), 10–27.

Patton, M. Q. (2002). Qualitative evaluation and research methods (3rd ed.). Thousand Oaks,

CA: Sage Publications, Inc

Stebbins, R. (2001). Exploratory Research in the Social Sciences. Thousand Oaks, CA:

Sage Publications

Stoll, L., Bolam, R., McMahon, A., Wallace, M. & Thomas, S. (2006). Professional learning

communities: a review of the literature. Journal of Educational Change, 7(4) 221–258.

doi: 10.1007/s10833-006-0001-8.

Swales, J. M. (1990). Genre analysis: English in academic and research settings.

Cambridge: Cambridge University Press.

University of Southern California. (2015). Organizing Your Social Sciences Research Paper:

Theoretical Framework. USC Libraries Research Guides. Retrieved:

http://libguides.usc.edu/writingguide/theoreticalframework

17

detwannabe.files.wordpress.com€¦ · web viewit was the researchers’ intent to investigate...

Documents