L I N D A S . B E H A R - H O R E N S T E I N , P H . D .D I S T I N G U I S H E D T E A C H I N G S C H O L A R A N D

P R O F E SS O R C T S I D I R E C T O R , E D U C AT I O N A L D E V E LO P M E N T

& E VA LUAT I O N C O L L E G E S O F D E N T I S T RY, E D U C AT I O N ,

V E T E R I N A RY M E D I C I N E & P H A R M AC YL S B H O R E N @ U F L . E D U

3 5 2 6 8 2 - 0 7 6 8

RESEARCH DESIGNS: AN OVERVIEWPRESENTATION TO THE HSC EDUCATIONAL

SCHOLARSHIP PROGRAM (ESP)SEPTEMBER 21, 2015

RESEARCH DESIGNS: AN ANALOGY

THIS PRESENTATION

• 1. An overview of qualitative, quantitative, and mixed methods research designs.

• 2. Methodological considerations for implementing the research design.

• 3. Several examples of research designs exemplified within UF-HSC educational research studies.

THINK ABOUT IT!

• 1. Consider an educational research study that you are doing, and the selected research design.

• 2. Reflect on an educational research study that you are planning, while still considering which research design is appropriate.

RESEARCH INQUIRY: WHAT DO I WHAT TO KNOW AND HOW CAN I FIND OUT?

• Do clinicians trained in team science do better than clinicians who are not trained in team science?

Researcher considerations• Study purpose• What data is needed

and how to collect it?

• How many participants?

• What research design should I use?

QUALITATIVE RESEARCH DESIGNS

NARRATIVE

• Explores shared culture, common experiences, or habits among a group of people.

• Uses a process of coding and categorizing data that relies on researcher’s skill and sophistication in coding and analyzing.

• Researcher perspective/previous experience must not bias interpretation of findings. Requires description of how researcher bias was managed or minimized.

HSC- NARRATIVE STUDIES

• Estrada A, et al. (under review). Incorporating interprofessional education into a veterinary medical curriculum.

•  • Spencer T, et al.( in prep). Assessment of

effectiveness of teaching shelter animal physical health using distance education technology.

• Behar-Horenstein LS et al. Perceptions of anticipated and actual Implementation among dental faculty trained in case-based learning J Dental Education. 2015: 79(9):1049-1060.

STEPS IN NARRATIVE RESEARCH

Identify a phenomenonthat addresses an educational problem

Purposefully select anindividual to learnabout the phenomenon

Collaborate withparticipant storyteller in all phases of research

Restory or retellthe individual’sstoryCollect stories from

the individual that reflect personal experience

Have themtell story

Collect otherfield texts

Build in past, present, future

Build in placeor setting

Validate the accuracy ofthe report

Write a story/summary about the participants’ personaland social experiences

Describe theirstory

Analyze story for

themes

Creswell (2012). Educational Research, 4th. ed. Boston: Pearson

GROUNDED THEORY

• Provide an explanation of process when theories are unavailable to address the research problem.

• Analyze datasets individually and iteratively using open coding to identify relevant categories and properties. Determine when data saturation has been reached.

• Findings are not generalizable. Subjectivity of data leads to difficulty establishing creditability and dependability of findings.

GROUNDED THEORY CONTINUED

• Results in construction of theory grounded in the data and research’s interpretation of the findings.

• EX: Perceptions of a new model of clinical dental education using two faculty and two student groups to ascertain their experiences.

• Findings showed individual, structural and resource-related issues that advanced and restricted the successful implementation of the initiative.

GROUNDED THEORY CONTINUED

Exchange of ideas and potential for fostering development of critical thinking skills, the use of modeling in multidisciplinary clinics, and just in time consultation were program benefits.

Reliance on attaining competencies to determine student skill development, patient allocation to student dentists, and a insufficiency of patients to perform needed procedures were drawbacks.

HSC- GROUNDED THEORY STUDIES

• Behar-Horenstein LS, et al. Serving the underserved: developing cultural competence among dental students. Journal of Dental Education 2015; 79(10):1189-1200.

• Behar-Horenstein LS, et al. Factors that advance and restrict programme change and professional development in dental education. Professional Development in Education 2012: 39(1): 65-81 DOI:101080/194152572012692701.

ETHNOGRAPHY

• To describe, analyze, interpret, and perhaps critique culturally-informally group’s shared behavioral patterns, beliefs, and language.

• A long-term commitment to field observations and interviews including in-depth study of culturally informed behaviosr, habits, and ways of life.

• Time consuming, reliant on observations of few individuals. Conclusions subject to researcher-bias.

CASE STUDY

• Examines one unit of analysis in a system bound by context and time, by using observations, recorded field notes, interview and survey findings.

• EX: Summarize the current practice of training medical students in palliative care issues.

• Can investigate complex social interactions that consist of multiple variables to render in-depth understanding of practice, training, or policy.

• Time, resources, lack of reliability, lacks validity and generalizability, labor intensive.

HSC-CASE STUDY

•Behar-Horenstein LS et al. A case study examining classroom instructional practices at a US dental school. J Dental Education 2005;69(6):639-648.

PHENOMENOLOGICAL

• Explores individuals’ experiences to describe the essence of a phenomenon.

• EX: Investigate homeless people’s perception of oral health and perceived consequences of dental treatment.

• Need to recruit participants, develop/pilot test interview guide, calibrate interview technique, and conduct in-depth individual interviews.

PHENOMENOLOGICAL, CONTINUED.

Look at change over time.

Develop understanding of how individuals construct meaning from their particular experiences.

Time, resource intensive, low credibility.

Reaching an endpoint is questionable and difficult.

ACTION RESEARCH

• Used to seek solutions experienced by individuals working in teams or within a community of practice.

• EX: Learn how researchers could collaborate with rural Bolivians to create access to community-based care clinic.

• Conducted in naturalistic settings, solution-focused.• Time, resource dependent. Lacks generalizability.• Systematic rigorous implementation may be lacking

by non-researchers.

Creswell (2012). Educational Research, 4th. ed. Boston: Pearson

STUDYING A PHENOMENON WITH INTENT TO MAKE CHANGE IN PRACTICE

Analyze andInterpret Data

Develop an Action Plan

Collect Data

Identify anArea of Focus

Creswell (2012). Educational Research, 4th. ed. Boston: Pearson

ACTION RESEARCH DESIGNS

Action Research

ParticipatoryPractical

•Studying local practices Involving individual or team- based inquiry•Focusing on teacher development and student learning•Implementing a plan of action•Leading to the teacher-as-researcher

•Studying social issues that constrain critical thinkingEmphasizing “equal” collaboration•Focusing on “life-enhancing changes”•Resulting in the emancipated researcher

ADAPTED META-SYNTHESIS WITH NARRATIVE SUMMARY, CONTINUED

• EX: Explored quantitative emancipatory research studies published between 2000-2011, an approach to research inquiry that minimizes the potential for those who are minoritized, remain voiceless, or are marginalized to characterize the similarities in findings.

ADAPTED META-SYNTHESIS WITH NARRATIVE SUMMARY

• Aim of critical appraisal analysis is to report practical outcomes across a body of work that share common over-arching elements.

• Conduct a systematic critical appraisal of the articles that meet search criteria.

• Due to heterogeneity of data, meta-analysis was not possible, so researchers presented narrative summary to present an overview essential findings related to the quantitative studies.

ADAPTED META-SYNTHESIS WITH NARRATIVE SUMMARY, CONTINUED

• Extract key findings and assigned them into groups where they were logical commonalities.

Synthesis 1: Agency in Community Health• The articles referred to agency in community health.

This synthesis included one category, building with three studies.

• 3.1.1 Category 1: Building Effective Community Health Service

• Findings from the studies suggested that in order to adequately serve community member health needs, people from different organizations worked cooperatively and responded in a culturally sensitive manner to individuals with health problems.

HSC- ADAPTED META-SYNTHESIS

•Behar-Horenstein LS, Feng X. Emancipatory research: a synthesis of quantitative evidence. ISOR. Journal of Research & Method in Education. 2015;5(3): 46-56.

QUANTITATIVE RESEARCH DESIGNS

EXPERIMENTAL (RANDOMIZED, CONTROL TRIAL)

• Intended to establish cause/effect between independent and dependent variables.

• Requires random assignment, use of covariates to control for potential group differences, pre-test/post-test.

• Considered one of the strongest designs for ensuring internal validity and generalizability. Good for clinical trials.

• Infeasible for educational research, ethical dilemmas when some reach treatment/intervention, others do not.

HSC-EXPERIMENTAL STUDY

• Johnson M, et al. (under review)Effectiveness of a new cadaveric teaching model for performing arthrocentesis. • within-subjects experimental design

QUASI-EXPERIMENTAL

• Studies intervention impact when random assignment is not possible.

• EX: Compared conventional medical follow-up to an approach that included additional nursing consultations regarding psycho-social adjustment and health–related quality of life issues among head/neck cancer patients.

QUASI-EXPERIMENTAL

• Identify and study plausible threats to validity.• To reduce plausibility of alternative causal

explanations add more observations at pre-test time points, more control groups.

• Several types of design types described in the literature.

• Feasibility depends on availability of good archived outcomes and ability to gather original data.

SHADISH, COOK & CAMPBELL

TIME SERIES

• Study intact group on specified variables over time before and after an intervention.

• Causal impact of treatment will be evident by different level or slope in post-intervention compare to pre-intervention measures.

• Observations not necessarily independent of one another. Inaccuracy of archived data.

SINGLE SUBJECT

• Study one individual before and after an intervention. Use individuals as own controls to establish utility of specific intervention.

• EX: Evaluated impact of eye movement desensitization reprocessing (EMDR) for patients suffering from PTSD.

• Carry-over effects, order and irreversibility as well as ethical problems when withdrawing treatment.

CORRELATIONAL

• Study extent to which two or more variables are associated and identify if one or more predictor variables forecast an outcome.

• Does not apply cause and effect.

• Analyzes relationships among large number of variables and provides information about degree of association to determine statistical significance of the degree of variance.

HSC-CORRELATIONAL STUDIES

• Delgado, et al. Are all dentiform teeth with simulated caries the same? A 6 years retrospective study in preclinical operative dentistry. (Accepted for publication). J Dental Education.

 • Echeto, LF, et al An evaluation of team-based learning and

traditional instruction in teaching removable partial denture concepts Dental Education 2015;79(9:1040-1048.

 • Behar-Horenstein LS, Garvan CW. Relationships among

knowledge, efficacy and practices instrument, color-blind racial attitudes scale, Deamonte driver, and defining issues test 2. (Accepted for publication). J Dental Education.

 • Behar-Horenstein, LS, et al . An assessment of faculty and

dental student decision-making in ethics. J American College of Dentists 2014; 81(4): 44-50. 

MEASURES OF EFFECT FOR DETERMINING PRACTICAL

SIGNIFICANCEStatistic Name

Symbol

Use

Effect size ES or d

Assess ES via significant t test result.

Eta squared

η2 Assess ES via significant ANOVA test.

• Determines if association is small (up to .02.), medium(.03 –.50), or large (over .50).

©2012 Linda S. Behar-Horenstein

MEASURES OF THE STRENGTH OF RELATIONSHIPS

Statistic Name Symbol Use Interpretation Example

Coefficient of determination

r2 Assess the proportion of variability in one variable accounted for by a 2nd variable.

Value is percentage of variance in one variable that is explained by a second variable.

r2 = .49 means time spent studying explained 49% of the variability in the final score.

R squared R2 Assess the proportion of variability in the DV accounted for by the combination of IVs in the regression equation

Same as above. R2 = .34, time on task, motivation, and prior achievement predicted 34% of the variability in student learning.

©2012 Linda S. Behar-Horenstein

PRE-EXPERIMENTAL

• Present treatment or intervention and measure impact at one point in time.

• EX: Assessed whether smokers responded differently to oral health message that emphasized the benefits of quitting smoking versus risks of continued smoking.

• Provides baseline information.• No control group, lacks power, findings are not

generalizable.

POSTHOC

• Explore outcomes from in vitro studies.

• EX: Evaluated bond strength of agents recommended for luting fiber-reinforced composite posts to root canal dentine before and after thermo cycling.

• Analyzed the effect of the luting agent, thermo cycling and region inside root canal on bond strength using ANOVA and post hoc Tukey tests.

• Precise laboratory testing removed influence of human variables such as responses to treatment.

• Lead to questions of similarity between lab findings and actual patients.

SECONDARY ANALYSIS

• Study the predictability of variable on dependent measures. Great for large datasets that use structural equation modeling. Can interchange independent and dependent variables.

• Relies on accuracy of data entry as well as someone else who collected and recorded data.

• Requires defining variables, variable types, dealing with missing data through imputation, ruling out mullticollinearity and satisfying model estimation parameters.

SURVEY

• Used to describe trends, beliefs or attitudes at one point in time, or over time.

• Easy to collect data from large samples and is cost efficient.

• Self-report data subject to social desirability bias. Researchers may make unfounded assumptions about the sample.

HSC –SURVEY STUDIES

 

• Connor BJ, et al Comparison of two clinical teaching models for veterinary emergency and critical care instruction. Accepted to the J Veterinary Medical Education.

• Estrada A, et al. (under review). Creating culture and community in the veterinary curriculum – using a student-centered orientation program.

HSC–SURVEY STUDIES

• Fernandez M, et al. Digital denture fabrication in predoctoral and postdoctoral education: a survey of US dental schools. J Prosthodontics. 2015;1–8.

• Rey, R et al. A preliminary study of curriculum time compared to clinical procedures in amalgam and composite posterior restorations. J Dental Education 2015;79(3):331-336.

• Normann S, et al. Perceptions of poisoning severity (in prep)

• Behar-Horenstein, LS et al. Dental school administrators’ attitudes towards providing support services for LGBT-identified students. J Dental Education 2015;79(8):965-970.

 

META-ANALYSIS

• Combines results from different studies on a particular topic/subject in hope of identifying patterns.

• Requires data extraction and assessing heterogeneity across findings.

• EX: Synthesized empirical studies designed to promote measurable changes in students’ critical thinking skills using instructional interventions. There were statistically significant, small average effect size, and heterogeneity among studies. Exploring predictors of the variance across effect size revealed that student major field of study and treatment length explained some of the variability among treatment effects.

HSC-META-ANALYTICAL STUDY

•Niu L, Behar-Horenstein LS, & Garvan C. Do instructional interventions influence college students’ critical thinking skills? a meta-analysis. Educational Research Review 2013: http://dxdoiorg/101016/jedurev201212002.

MIXED METHODS RESEARCH DESIGNS

MIXED METHODS

• Capitalizes on strengths of quantitative and qualitative inquiry when one type of approach is insufficient.

• Challenges – time consuming, labor intensive. Need to show how use of both methods enhanced understanding of answers to research questions.

KEY DECISIONS IN CHOOSING A MIXED METHODS STUDY

• The level of interaction between the quantitative and qualitative strands.

• The priority of the strands.• The timing of the strands.• Where and how to mix the strands.

VISUAL DIAGRAMS OF MM DESIGNS

I. Convergent Mixed Methods Design

II. Embedded Mixed Methods Design

QUAN(Data and Results) +

QUAL(Data and Results)

Interpretation

QUANData and Results

QUAN(Data and Results)

Interpretation

III. Explanatory Mixed Methods Design

IV. Exploratory Mixed Methods Design

QUAN(Data and Results)

QUAL(Data and Results)

qual(Data and Results)

quan(Data and Results)

Follow-up

Building

VISUAL DIAGRAMS OF MM DESIGNS (CONT’D)

MIXED METHODS - SOME EXAMPLES

• Found statistically significant differences using EFA and MANOVA in two student reflective writing assignments. Explained underlying differences using qualitative examples.

• A follow-up study revealed similar statistical findings. While implementing a QUAN→qual sequential mixed method design, researchers used a deductive thematic analysis and explored those differences contextually using qualitative findings.

HSC-MIXED METHODS STUDIES

• Behar-Horenstein LS. et al (under review). Dental students’ expression of cultural competence.

• Isaac, C. et al. - Impact of interviews on dental students' expressions of cultural competency. J Dental Education 2014; 79(3): 312-321.

APPLYING WHAT YOU KNOW

• 1. Describe an educational research study that you are doing, and its research design. How do you know that this design is appropriate to the study’s aims?

• 2. Describe an educational research study that you are thinking about doing. Which research design seems appropriate?

L INDA S . BEHAR-HORENSTE IN , PH .D .D IST INGUISHED TEACHING SCHOLAR AND

PROFESSOR CTS I D IRECTOR , EDUCAT IONAL DEVELOPMENT & EVALUATION

COLLEGES OF DENT ISTRY, EDUCAT ION, VETER INARY MEDIC INE & PHARMACY

LSBHOREN@UFL .EDU352 682 -0768

RESEARCH DESIGNS: AN OVERVIEWPRESENTATION TO THE HSC EDUCATIONAL

SCHOLARSHIP PROGRAMSEPTEMBER 21, 2015