· web viewpacific grove, ca: brooks/cole. author technical support services created date...

1

MSW Evaluation Report: 2013-2014Submitted by: W. Jay Gabbard, MSW, Ph.D., WKU MSW Program Assessment CoordinatorTo: Dr. Dean May, Department Head-Social Work-Western Kentucky UniversityDr. Amy Cappiccie, MSW Program DirectorDate: 6/16/2014

Contained in this report is a summary of evaluation data gleaned from the MSW program for the academic year 2013-2014. This information will be utilized to supplement departmental and college reports, as well as for CSWE Accreditation purposes. As well, these results will be shared with the faculty and larger social work community, in order to continually receive input and improve the quality of the MSW program at WKU.

ACAT

In order to assess the mastery of foundation level knowledge of MSW students at WKU, the ACAT assesses standard curricular areas within accredited graduate social work programs in the United States. Curriculum A, the one administered to “traditional” WKU MSW students, assesses knowledge in the following curricular areas:

Human Behavior in the Social Environment Social Policy Social Work Practice Research Methods Diversity Populations at Risk Social and Economic Justice Values and Ethics

Results from the ACAT are provided to individual students in the form of standard scores for each curricular area as well as for overall performance for the cohort taking the test. These curricular areas and the standard scores associated with them can be linked directly to current program objectives.

The ACAT specifically assesses the following explicit curriculum standards from the 2008 CSWE Educational Policy and Accreditation Standards (EPAS): 2.1.1-2.1.10.

2

ACAT Results-2 Year Traditional Cohort (Class of 2015)

As is done annually, the ACAT was administered to WKU MSW students at the end of their foundation year of courses on April 26, 2014. The exam is carefully proctored and traditional students are required to sit for the exam. Below are the results of this latest administration:

Area Standard Score %ile Reference Group Size

Diversity 490 46 7187

Populations at Risk 504 52 7187

Social/Economic Justice 518 57 7187

Values and Ethics 531 62 7187

Policy & Services 536 64 7750

Social Work Practice 575 77 7750

HBSE 499 50 7750

Research Methods 567 73 7750

OVERALL PERFORMANCE 539 65 7187

The results of this administration indicate that overall, this cohort of traditional MSW students performed at the 65th percentile compared to similar MSW students nationwide. Thus, per the official ACAT report, based on a comparison group of 7187 MSW students taking the ACAT with 8 areas evaluated, 65% would be expected to score at or below WKU’s overall performance score and 35% would be expected to score higher.

3

Per the 2013-2014 WKU MSW score report, “ACAT scores range from 200 to 800 with an average of 500 and a standard deviation of 100. Nationally, 68 percent of the scores in any given year fall between approximately 400 and 600. Year to year variations in the size of the reference groups will cause scores to fall outside these limits. The content area scores are compared with a reference group of other examinees taking the same content area. The overall performance score is compared with other examinees taking the ACAT in this discipline with the same number of content areas.”

Students in this cohort scored highest in the areas of social work practice and research methods, just as they have consistently for most of the ten years the test has been administered at WKU. They also fared well in the areas of Policy & Services and Values & Ethics, two areas where they have traditionally scored lowest. The lowest mean scores were in the domains of diversity and HBSE. The diversity results were interesting, given the heavy infusion of diversity content in foundation courses and flagship diversity course required during the first semester. However, these results will be reviewed at a faculty retreat (see below) to see if additional diversity content needs to be added or modified in the curriculum.

In terms of overall performance, scores increased 6 points since the last administration in 2013. The areas of greatest growth were in Values & Ethics (plus 13 percentile points) and Social Work Practice (plus 11 percentile points). The increase in the Ethics domain is notable due to the fact scores in this area have often been the lowest with foundation level MSW students at WKU. Practice has usually been the highest rated domain on the ACAT with our students, so this additional increase in scores is auspicious.

The areas of most significant decline were in Diversity (minus 8 percentile points) and Social and Economic Justice (minus 5 percentile points). Diversity has traditionally been one of the lowest rated domains on the ACAT with our foundation students, even though they consistently (see below) manifest statistically significant gains in cultural self-awareness on the Lum’s inventory.

Interestingly, one student in this foundation cohort attained the highest overall percentile score ever in the MSW program (96th percentile) while another earned one of the lowest scores ever (10th percentile). Percentile wise, this group’s performance was one of the highest in the history of the ACAT at WKU, which is certainly a positive development.

As in prior years, the results of this administration will be discussed at the MSW Faculty Retreat on August 20, 2014 and a strategy devised to maintain strengths and address perceived deficiencies in the aforementioned areas. As well, ACAT results will be shared at the next Departmental Advisory Committee (DAC) meeting during the 2014-2015 academic year, in order to solicit feedback from community members and stakeholders on how to improve in weaker areas. This helps close the feedback loop with a group heavily vested in the success of the program.

4

Individual results were also provided to students, in order that they might assess areas of strengths and weaknesses as they move forward into their concentration year of studies and prepare for the Practice Readiness Exam (PRE) exam. Their academic advisors play an integral role in this process.

Lum’s Cultural Awareness Inventory

One of the primary goals of the MSW program at WKU is that graduates emerge with the knowledge, skills, and self-awareness required for culturally sensitive practice (EPAS 2.1.4; Gabbard, Starks, Cappiccie, & Jaggers, 2011). Towards this end, both advanced standing and traditional students complete flagship diversity courses (SWRK 612 or SWRK 501) to master diversity and cultural competency content. This material is also infused throughout the curriculum and in many electives such as Homelessness, Forensic Social Work, Expressive Therapies, and Alternative Therapies. In order to assess student’s cultural self-awareness, the MSW program at WKU utilizes an instrument that has been validated and employed in numerous diversity evaluative studies nationwide.

The Cultural Awareness Inventory is adapted from Lum’s Social Work Cultural Competencies Self-Assessment (2003). The Cultural Awareness Inventory consists of a number of sections, including an introduction, background information, 44 statements that are answered on a 4-point scale, and two open-ended items at the conclusion of the instrument.

The majority of the instrument is made up of the 44 self-reported statements that encompass the cultural domains of: cultural awareness, knowledge acquisition, skill development, and inductive learning. Students respond to each statement based on the following directions:

Rate yourself on your level of competency on a scale of 1 – 4:

1 = Unlikely 2 = Not very likely 3 = Likely 4 = Definitely

Circle the appropriate number.

5

Given EPAS 2.1.4 and the MSW program’s stated mission, “To educate and prepare students for professional social work practice to meet the needs of increasingly diverse rural populations,” the Cultural Awareness Inventory is used to assess a critical component of this program standard and mission. This instrument measures self-perception of cultural awareness while two additional instruments (ACAT and PRE) assess more concrete diversity concepts learned throughout the program. It is this combination of knowledge, skills, and self-awareness that constitutes the preparation of culturally competent social work practitioners (Gabbard, Starks, Cappiccie, & Jaggers, 2011). The program wants to produce graduates who know the traditions, histories, and cultural norms of multicultural groups and can put this knowledge into practice by designing culturally sensitive interventions.

Cultural Awareness Inventory-How the Instrument and Data are Used:

The Cultural Awareness Inventory is employed as a pre-and post-test assessment procedure, statistically evaluated with dependent t-tests. It is administered at the beginning (orientation) and conclusion of the student’s MSW program. The goal is to assess for significant changes (or lack thereof) in students’ perceived level of cultural competency and cultural self-awareness. While self-perceived change does have its obvious limitations (diversity content is also measured more concretely on the ACAT and PRE), it does allow for students to reflect back on the knowledge and skills they have acquired in courses and in fieldwork, which is a valuable exercise for them to engage in at the conclusion of their study. This, coupled with a self-reflection paper they write in their flagship course, helps them to identify biases, stereotypes, and schemas they must come to grips with in serving multicultural (often oppressed) clients.

Results-Class of 2014 (Advanced Standing and Traditional)

Below are descriptive statistics on all individuals who completed the pre and post-tests in the 2015 graduating class. Following this data are dependent t-test results of this inventory for advanced standing and traditional graduates in 2014. While the scores are approximately normally distributed (per the primary assumption of parametric tests), a nonparametric Wilcoxon Sign test was also run for comparative purposes, given the presence of some outliers and relatively low correlation between scores. It is important to note there were a number of students who did not complete instruments, who were not present for the pre-test and/or post-test when they were administered or who did not enter the last four digits of their student number (for pre-post comparisons) who are not reflected in the data below.

6

Descriptive Statistics

N Range Minimum Maximum Mean Std. Deviation Variance

age 36 30 22 52 32.89 8.834 78.044

pre cc score 36 56 113 169 137.61 13.217 174.702

post cc score 36 63 113 176 155.44 14.038 197.054

Valid N (listwise) 36

gender

Frequency Percent Valid Percent Cumulative

Percent

Valid

male 5 13.9 13.9 13.9

female 31 86.1 86.1 100.0

Total 36 100.0 100.0

ethnicity

Frequency Percent Valid Percent Cumulative

Percent

Valid

european american 23 63.9 63.9 63.9

african american 10 27.8 27.8 91.7

latino american 2 5.6 5.6 97.2

asian american 1 2.8 2.8 100.0

Total 36 100.0 100.0

The average age of respondents was 33 years old, with a range of 30 years and a standard deviation of 9 years. Similar to the demographic constitution of the MSW program, most of the respondents were female. In terms of ethnographic makeup, the majority of respondents were Caucasian, followed by African-American, Latino, and Asian-American respondents.

7

Paired Sample t-Test Results (2014 Graduates)

Paired Samples StatisticsMean N Std.

DeviationStd. Error

Mean

Pair 1pre cc score 137.61 36 13.217 2.203post cc score 155.44 36 14.038 2.340

Paired Samples CorrelationsN r Sig.

Pair 1pre cc score & post cc score

36 .286 .090

Paired Samples TestPaired Differences t df Sig. (2-tailed)

Mean Std. Deviation Std. Error Mean

95% Confidence Interval of the Difference

Lower Upper

Pair 1pre cc score - post cc score

-17.833 16.295 2.716 -23.347 -12.320 -6.567 35 .000

8

Looking at the results of the dependent t-test for the 2014 graduates (both traditional and advanced standing) who completed both the pre and post ratings, there was a significant (at the .01 level) increase in MSW students’ cultural competency scores from when they started the program at orientation to the end of their course of study. In terms of correlation between the two variables, it was not significant at the .05 level (r =.90) and displayed a low, positive relationship between the variables. Due to the low correlation between the scores and presence of some outliers in the data, a Wilcoxon Sign test (the nonparametric equivalent of the dependent t test) was also run, for comparative purposes. This test also a significant result at the .01 level.

The average increase of 18 points on the inventory yields evidence that students (from their perspective) became more sensitive to culturally diverse groups and skilled in providing culturally sensitive interventions in practice during their time in the MSW program at WKU. This is an important finding, as one of the primary goals of the social work profession and MSW program at WKU is to produce graduates who are highly skilled in assessing and intervening with diverse client populations.

Next are displayed the results and a graph of the dependent t-test for 24 advanced standing students in the graduating 2014 cohort who completed the pre and post Lum’s assessment.


DeviationStd. Error

Mean


9

Paired Samples CorrelationsN Correlatio

nSig.


24 .141 .510




Lower Upper


-19.250 15.674 3.199 -25.869 -12.631 -6.017 23 .000

The results of the sample of students in the advanced standing cohort’s dependent t-test also indicated a statistically significant (at the .01 level) increase in self-reported cultural sensitivity from orientation to graduation. The average increase for this cohort (19 points) was higher than the overall gain for the combined graduating class. Similar to the combined results, the correlation between the two variables was not significant at the .05 level and low positive. The results of the nonparametric Wilcoxon Sign test yielded similar results with a significant difference between the scores at the .01 level.

10

The significant increase in cultural competency for advanced standing students supports the program and professional goals of producing graduates who are sensitive to the needs of their diverse clients and skilled in providing culturally sensitive interventions. The infusion of diversity content throughout the curriculum and flagship diversity courses (for these students, SWRK 612), contributes to students’ continual growth in these areas while in the program. Finally, below are the dependent t-test results for 12 students in the 2 year traditional cohort who graduated in 2013 (pretest was obtained at orientation in 2012). Please note that there were a number of students who did not complete the pre or post-test, did not enter the last four digits of their student number for comparison purposes, or who left the program (or shifted to part-time status), which contributed to lower completed pre/post numbers with this cohort.


DeviationStd. Error

Mean


Paired Samples CorrelationsN Correlatio

nSig.


12 .303 .338




Lower Upper


-15.000 17.833 5.148 -26.330 -3.670 -2.914 11 .014

11

This sample of traditional program graduates manifested a lower average increase in cultural competency scores (15 points) than their advanced standing colleagues and the combined class result that was statistically significant at the .05 level.

Similar to the overall and advanced standing correlation results, the r value was statistically insignificant at the .05 level and low positive. Per the Hypothesis Test Summary displayed above, the nonparametric Wilcoxon Sign test also yielded a statistically significant result at the .05 level. However, the fewer number of individuals in this cohort should be taken into consideration when evaluating the results of these tests.

Historically, in past graduating cohorts, there has been a more pronounced increase in cultural competency scores for traditional students (compared to advanced standing ones) that might have to do with the fact they take more courses with diversity content in them at the graduate level. As well, their flagship diversity course (SWRK 501) is solely dedicated to cultural diversity and cultural competency. As well, many of these students may have (unlike Advanced Standing students) had less exposure to these concepts in their undergraduate degree programs, so their gains in this area would likely be more pronounced than those with a prior history of exposure to diversity in BSW programs.

However, this cohort was atypical in the fact that they actually displayed a notably lower increase in self-awareness scores. Interestingly, two individuals in this traditional cohort actually rated lower on their post-scores and one rated no change at all. This is definitely an atypical finding that should be noted. It will also be discussed at the MSW retreat in Fall 2014 in order to assess if any additional considerations need to be considered with this cohort.

In sum, students in both graduating cohorts, as well as the overall group, indicated a statistically significant (at the .01 and .05 levels) increase in self-reported cultural awareness during their tenure at WKU. These results indicated a degree of success in reaching a stated goal of the MSW program, with regards to cultural competency and diversity:

12

“More specifically, dependent t-tests (students put the last few digits of their student id numbers on the instruments so pre-post results can be paired) after graduation each year. An increase in overall positive trend, confirmed by a p value of less than benchmark standard of .05 (traditional in social science research) is the rejection level for statistically significant findings and an indication of success in this area.”

While the presence of intervening variables (e.g. additional diversity trainings, familiarity with the test, history) likely influenced these results, it appears that students in the 2014 graduating class perceived themselves as more culturally competent at the end of the program than when they matriculated. The fact this trend had consistently been maintained over the life of the MSW program at WKU is evidence that faculty are doing an adequate job of preparing these students for culturally sensitive practice.

PRACTICE READINESS EXAM (PRE)

Per departmental and WKU Graduate School requirements, all graduating MSW students must successfully pass a Practice Readiness Exam (PRE) during their final semester in order to successfully complete the program. Per the 2013-2014 MSW Student Handbook, the PRE exam gives students the opportunity to demonstrate basic competency in essential content areas of social work practice with an emphasis in rural settings. The exam contains 100 multiple choice questions in an objective format, similar to social work licensure exams. It is indeed an excellent tool to prepare students to pass the Intermediate licensure exam post-graduation, one of the desired outcomes of the MSW Program at WKU. While the MSW Program does not publish licensure pass-rate statistics, historical and anecdotal evidence points to a considerable degree of success in this area, at both the CSW and LCSW levels.

Students are required to pass the PRE exam with a score of 70 or better in order to graduate from the program. Per the WKU Graduate School Policy, a student only has two attempts to pass this comprehensive exam. Consequently, a student who fails the exam two times is subject to dismissal from the MSW Program. However, it should also be noted that students who fail the exam are provided with written information regarding their performance and advised on areas they need to improve on before they take it (in several weeks or during the next Fall semester).

As the test assesses knowledge of specific content from readings, lectures, and field practicums/experiences, it is considered an important evaluative measure of student learning and retention in the program. It measures not only concrete concepts gleaned but also puts students in “practice situations” where they must respond based on their professional and ethical training. It has been continually refined over the 10 years it has been administered and an item analysis is conducted after each administration to assess the efficacy of specific questions (to decide which ones to keep and which to discard). New questions are continually developed to have fresh questions on each administration, in addition to “battle tested” ones.

13

It should also be noted that, after evaluating the evaluation strategy employed in years past, a decision was made this year to drop the self-rated foundation and concentration level objective tool in lieu of a more concentrated evaluation of the PRE, which is anchored in conceptual knowledge and the actual ability of students to evaluate practice and ethical situations. In the future, even more complex analyses will be conducted with PRE results and they will be correlated with other measures such as the ACAT and Lum’s Cultural Competency Inventory, in order to assess program outcomes.

The 2014 exam included questions in the following subject areas (numbers of questions on each area exam are denoted): Cultural Competency and Diversity(4) Foundation (27) and Advanced Practice (16) Human Behavior in the Social Environment (7) Ethics (6) Foundation (10) and Advanced Social Welfare Policy (4) Groups (6) Diagnosis (5) Family Practice (6) Administration and Supervision (4) Community Organization (6) Rural Communities (2) Research (7)

Approximately 67 students took the PRE on April 7, 2014. Of these, 59 students achieved a passing score of 70 and 8 failed to achieve this benchmark (following item analysis). Thus, approximately 88 percent of students taking the exam passed and 12 percent failed during this administration. These numbers were similar to prior success rates on the exam over a ten-year period, which usually average around 10 percent needing to do a retake.

The subject areas where the most students appeared to struggle (taking into account the number of questions in each area) were: Foundation Policy, Diagnosis, and Community Organization. The areas students missed the fewest questions in were: Ethics, Advanced Practice, Research, and Foundation Practice.

The first result is interesting given the fact traditional students in this graduation cohort tended to score only average on Foundation Level Policy questions on the ACAT. Of course, as mentioned above, there was a marked improvement in this subject area with students taking the ACAT this year (Class of 2015). While these two tests are not identical in terms of content, it is valuable to look at trends across evaluative measures.

14

Some of the areas where students fared the best (Foundation Practice and Research Methods) were perhaps not coincidentally the same two areas that foundation students usually perform the best in on the ACAT. For years, these have been the two domains where students have scored the highest in on the ACAT and this pattern is usually also present on the PRE.

Students have also traditionally performed well on the Advanced Practice questions on the PRE. Interestingly, they have usually scored low in the Ethics domain on the ACAT but they performed exceptionally well in this area on the PRE. Scores also improved dramatically in this domain on this year’s ACAT results, indicating a shift in student performance.

Similar to the ACAT and Lum’s Inventory, the results of the PRE will be reviewed at the MSW Program Retreat in August 2014. An additional factor that will be discussed is whether to offer the test on a computer in future administrations, which would even further mirror the process with the CSW and LCSW exams. As well, the PRE committee will review the efficacy of the questions from this year and create new ones for the 2015 test.

MSW FIELD DATA/EVALUATION

Just as with the domains assessed above (and in accordance with the accreditation standards established by the Council on Social Work Education (CSWE)), the MSW Program annually collects field data, in order to assess the program’s compliance and success in meeting established benchmarks for each standard. More specifically, the percentage of students achieving each CSWE competency is calculated for both traditional and advanced standing graduates. This information, which is gathered by the Field Director and posted on the MSW Program website, helps the program evaluate areas of strength and potential growth in field objectives. Below is a summary from May 2014 summarizing the field assessment for academic year 2013-2014:

ASSESSMENT OF STUDENT LEARNING OUTCOMES

WESTERN KENTUCKY UNIVERSITY MSW PROGRAM

LAST COMPLETED ON MAY 2014

This form is used to assist the COA in the evaluation of the program’s compliance with Accreditation Standards stated below.

4.0.2 The program provides evidence of ongoing data collection and analysis and discusses how it uses assessment data to affirm and/or make changes in the explicit and implicit curriculum to enhance student performance.

4.0.5 The program appends the summary data for each measure used to assess the attainment of each competency for at least one academic year prior to the submission of the self-study.

15

All Council on Social Work Education programs measure and report student learning outcomes. Students are assessed on their mastery of the competencies which comprise the accreditation standards of the Council on Social Work Education. These competencies are dimensions of social work practice which all social workers are expected to master during their professional training. A measurement benchmark is set by the social work programs for each competency. An assessment score at or above that benchmark is considered by the program to represent mastery of that particular competency.

COMPETENCYCOMPETENCY BENCHMARK PERCENT OF STUDENTS ACHIEVING BENCHMARK

% OF STUDENTS ACHIEVING BENCHMARK

(TRADITIONAL)

% OF STUDENTS ACHIEVING BENCHMARK

(ADVANCED STANDING)Identify as aprofessionalsocial worker and conduct oneself accordingly.

70% of students will earn 4 or higher

98% 98%

Apply social work ethicalPrinciples to provide professional practice.


80% 95%

Apply criticalthinking to inform and communicate professional judgments.


73% 93%

Engage diversity and difference in practice.


80% 86%

Advance humanrights and social

70% of students will earn 4 or 80% 87%

16

and economic justice.

higher

Engage in research-informed practice andpractice informed-research.


80% 84%

Apply knowledge of human behavior and the social environment.


86% 95%

Engage in policy practice to advance social and economic well-being and to deliver effective social work services.

70% of students will earn 4 or higher 86% 86%

Respond to contexts and shape practice.


86% 84%

PracticeEngagement


86% 95%

PracticeAssessment


86% 95%

PracticeIntervention


86% 95%

PracticeEvaluation


86% 95%

Looking at the results above, it is apparent that the MSW Program was successful during the 2013-2014 year in meeting all of the competency benchmarks established by the CSWE, across both program tracks. Traditional students rated particularly high on identification and conduct as a professional social worker and practice engagement, assessment, intervention, and evaluation.

17

Advanced standing students similarly rated high on identification and conduct. They also significantly exceeded the benchmark on applying ethical and HBSE principals in practice, as well as in practice considerations. One area that rated lowest for both groups was the research-practice linkage, which is continually stressed in the Research Methods and Applied Research courses. While students exceeded the benchmark in this area, perhaps new strategies can be devised to help students recognize and apply research concepts in the field/practice arena. This topic will be discussed at the Fall 2014 MSW Program Retreat.

References

Gabbard, W. J., & Starks, S., Cappiccie, A., & Jaggers, J. (2011). An effective model of teaching cultural competency to MSW

18

students in a global society. Journal of Human and Social Sciences, 6(2), 204-208.

Lum, D. (2003). Culturally competent practice: A framework for understanding diverse groups and justice issues. Pacific Grove, CA:

Brooks/Cole.

· web viewpacific grove, ca: brooks/cole. author technical support services created date...

Documents