Mixed Methods in Practice: Data visualisation for the evaluation

study 'Bildungsberatung Kärnten'Birgit Senft, Maria Helene Aichholzer-Pahr, Beate Gferer

Freelance evaluator, Bildungsberatung Kärnten

MQIC 2018 MAXQDA International Conference 28. February – 2. March 2018 | Berlin

Background and goals:

In the context of the participative and formative evaluation of the project 'Bildungsberatung Kärnten' (Educational

Counselling Carinthia), individual perspectives of clients and cooperation partners were collected in addition to

quantitative data and edited for the project team.

Key questions of the evaluation:▪ How do the cooperation partners perceive the project and how do they rate the cooperation?

▪ How satisfied are the clients with the counselling service?

▪ How effective is the counselling service? (from the clients‘ point of view)

Methods:

Stockmann, R. (2006). Evaluation und Qualitätsentwicklung. Eine Grundlage für wirkungsorientiertes Qualitätsmanagement. Münster u.a.: Waxmann.

Stockmann, R. (Hrsg.). (2007). Handbuch zur Evaluation. Eine praktische Handlungsanleitung. Münster: Waxmann.

IBM Corp. (2013) IBM SPSS Statistics for Windows, Version 24.0. [Computer software]. Armonk, NY: IBM Corp.

Tableau Software Inc. (2017) Tableau Desktop Personal Edition 10.4.0 [Computer software]. Seattle: Tableau Software Inc.

Mayring, P. (1993). Qualitative Inhaltsanalyse. Grundlagen und Techniken (4. Aufl.). Weinheim: Deutscher Studien Verlag.

VERBI Software. (2017). MAXQDA. Software für qualitative Datenanalyse, 1989 bis 2017. Berlin: Consult. Sozialforschung GmbH.

▪ The processing of text amounts of different sizes can be performed efficiently

▪ Visualisations with MAXQDA maps inspire reading

▪ One-case models are ideal for case supervision or intervision

▪ One-code models provide a good overview of a specific issue

▪ Besides the content of codes, frequencies are relevant for the evaluation of the process quality

▪ The data export by group or category allows for a quick validation of codes

▪ Codes can be transformed into variables, making cross-tabulations possible

▪ Large amounts of data can be reduced to clear forms of presentation

▪ The export option into Excel allows for further analyses, e.g. with statistical programmes (e.g. SPSS, tableau)

Contact: Birgit Senft, clinical psychologist, Master of EvaluationHöhenweg 1, 9020 Klagenfurt am Wörthersee Liechtensteinstraße 22/3/11, 1090 Viennawww.statistix.at

1

Table 1 Interview questions with codes and code frequencies by type of cooperation partner

Results of client interviews Results of cooperation partner interviews

Benefit of mixed methods and visualisations for the project References

2

21

2

Type of data/group Type of survey Count

Primary data: Survey of cooperation partners Telephone/ semi-structured interview N = 32

Primary data: Survey of participants of the competence Workshop Paper-pencil/ questionnaire N = 221

Primary data: Survey of clients Telephone/ semi-structured interview N = 40

Primary data: Potential clients Focus group N = 10

Secondary data: Clients characteristics Statistical analysis N = 2.538

Secondary data: Counselling sessions Statistical analysis N = 5.947

Data analysis:

Quantitative analyses: Statistical calculations/significance testing using SPSS 24

Graphs using tableau 10.4.0

Qualitative analyses: Qualitative content analysis with inductive category development, subsuming inMAXQDA 12.3.2 Development of types based on the categories - transformation ofthe code 'type' into a variable and cross-tabulation in MAXQDA, statistical analyses

Questions and codesactive cooperation

partners (N=17)

cooperation partners with little acitivity

(N=7)

inactive cooperation partners (N=8)

SUM(N=32)

1.) What is your spontaneous association when you hear Bildungsberatung Kärnten?Educational counselling 10 4 1 15something for everyone 1 1 2'competence' 3 1 4explicit mention of 'USP' 3 1 4mostly general comments 1 1 1 3little knowledge of the BBK 2 2 4personal contact 3 1 4other comments 4 1 2 7

// 1 question excluded

3.) What are typical clients that you refer/send to the Bildungsberatung Kärnten?specific clients 11 5 4 20people who want to (re)position themselves 10 10exclusion 3 3here for (almost) everyone 8 8people who have to reposition themselves 3 1 1 5few skills 1 1 2other comments 7 2 4 13

4.) What type of feedback have you received from your clients regarding the Bildungsberatung Kärnten?positive feedback 7 3 1 11little feedback 7 3 1 11other comments 5 4 9

5.) How do you perceive the cooperation with the Bildungsberatung Kärnten?

active cooperation 13 3 16little cooperation 1 3 4 8use of various possibilities 6 6promotion of autonomy of clients 4 1 5desire for exchange 3 1 3 7other comments 6 2 2 10no comment 1 1 2

SUM 149 46 41 236Average count of codes 8,8 6,6 5,1 7,4

Figure 3 Literal quotes for the code positive feedback

Figure 2 Codes for the question how the counselling situation was perceived by typology of clients

1, 2

3

4

6

5

4

2

3

1

5

6

Figure 1 Typology of clients

DataTriangulation

Methods

Developement and use of

questionnaires

Fokus groups

Online survey

Statistical analysis

Telephone interviews

Semi-structured interviews

Clients

Cooperation partners

Assessment of the team

Secondary data

Potential clients