EVALUATION OF THE CLINIC AND HOSPITAL PROJECT

Presented by Sabine Frielinghaus

Final Research Results 20 April 2015

Revised 23 April 2015

2

BACKGROUND

Milk SA is targeting the South African consumer market

(mainly LSM 3 – 7) at government clinics and hospitals in

various provinces.

The campaign is a series of educational presentations

focused on the health and nutritional advantages of dairy

products.

To ensure that Milk SA’s funds are being invested cost-

effectively, an initiative to track the effectiveness and

success of campaign was required.

MQ Market Intelligence was invited to conduct a mystery

evaluation of Milk SA’s Clinic and Hospital project.

3

THE OBJECTIVE

is to measure the success of the Clinic and Hospital Project.

MEASURES OF SUCCESS

Opinion and learning of the target

market (consumer) perspective

Effectiveness of implementation of the

initiatives

4

This is a field based research technique where independent, trained field workers pose

as consumers to gather specific information

about a product/service and to experience how it

feels to be a consumer of the product or service.

The Mystery Observer methodology is ideal for

evaluating how suitable and effective marketing

initiatives are according to the target market’s

response over time.

THE APPROACH

is to implement a MYSTERY OBSERVER STUDY

5

EXECUTION

A structured

questionnaire and

specific tasks are created

to guide the mystery

observer.

The observations take

place on selected

activation days

throughout the campaign

(except the first day as a

few teething issues might

need sorting out).

At the end of the

evaluation, the observer

asks the direct target

market (end consumers)

questions about their

experience and learning.

Professional

interviewers with at least

3 years of experience are

recruited. All fieldworkers

are personally trained by

their supervisor.

All service providers are

told that Milk SA will

randomly send mystery

observers to their

activations. The specific

event is not disclosed, so

the service providers are

highly incentivised to

perform at all times.

QUESTIONNAIRE OBSERVATIONS INTERVIEWS SERVICE PROVIDERS

1 2 3 4

6

NAME PROVINCE DATE

1 Protea Glen Clinic Gauteng 20/02/2014

2 Joubert Park Clinic Gauteng 21/02/2014

3 Jeppe Clinic Gauteng 21/02/2014

4 Kalafong Hospital Gauteng 06/03/2014

5 Edenvale Hospital Gauteng 07/03/2014

6 Nhlazatshe 4 Clinic Mpumalanga 15/04/2014

7 Tjakastad Clinic Mpumalanga 15/04/2014

8 Kwamaphumlo Clinic KZN 05/06/2014

9 Thafamasi Clinic KZN 06/06/2014

10 Sivananda Clinic KZN 11/06/2014

11 Themba Hospital Mpumalanga 18/06/2014

12 Calcutta Clinic Mpumalanga 23/06/2014

13 Matikwana Hospital Mpumalanga 23/06/2014

14 Martjie Venter Hospital Eastern Cape 17/07/2014

15 Cradock Hospital Eastern Cape 17/07/2014

16 Oukasie Clinic North West 23/02/2015

17 Mothotlung Clinic North West 23/02/2015

18 Oukasie Clinic North West 24/02/2015

LOCATION

18 clinics across South

Africa participated in the

evaluations.

7

IMPLEMENTATION

RATE

DEFECT RATE

Main presentation

78% 22%

Main presentation

without planning errors *

93% 7%

Specialised

presentation

without planning errors

43%** 57%

IMPLEMENTATION RATE

* Planning errors occurred when the activation did not take place at the

scheduled time without at least 4 hours notice. For example, the schedule

had the incorrect date / the activation started 2 hours early.

** This rate could be slightly higher as the specialised presentation could

have run at the same time as the main presentation with a different group,

however only one evaluator was on site and accurate measurement was

not possible.

8

IMPLEMENTATION RATE OF PROMOTIONAL MATERIAL

On average 58% of the promotional material was implemented correctly.

Is this roll-up banner up: ‘Have milk, maas and yoghurt’?

Is this roll-up banner visible to most people in the waiting area?

Is the roll-up banner up: ‘Have milk, maas and yoghurt’?

Is the roll-up banner visible to most people in the waiting area?

Is the A1 poster up: ‘Have milk, maas and yoghurt’?

Is the A1 poster visible to most people in the waiting area?

Is the blue Sharkfin banner up: ‘Have milk, maas and yoghurt’?

Is the blue Sharkfin banner visible to most people in the sitting area?

yes no

57%

50%

64%

71%

57%

64%

57%

43%

43%

50%

36%

29%

43%

36%

43%

57%

9

50

40

36

27

26

25

21

15

15

14

13

Oukasie Clinic

Jeppe Clinic

Sivananda Clinic

Edenvale Hospital

Average

Thafamasi

Kalafong Hospital

Cradock Hospital

Martjie Venter Hospital

Joubert Park Clinic

Nhlazatshe 4 Clinic

DURATION OF THE ACTIVATION (in minutes)

The duration of the activations varied considerably.

10

L1

L2

L3

L4

L5

L6

L7

L8

L9

L10

L11

L12

L13

L14

SCRIPT ADHERENCE - Laminates Shown

The average script adherence was 80%

93%

93%

71%

93%

79%

79%

86%

57%

57%

64%

86%

79%

93%

93%

7%

7%

29%

7%

21%

21%

14%

43%

43%

36%

14%

21%

7%

7% L7 - importance of variety of food

yes no

11

L1: Did the presenter explain what dairy is?

L2: Did the presenter give examples of dairy?

L3: Presenter mentioned that dairy acts as ‘fuel’ in the body?

L4: Did the presenter mention the 9 nutrients in dairy?

L5: Presenter emphasised that we all should eat dairy daily?

L6: Presenter explained the benefits of the nutrients?

L7: Presenter mentioned the importance of variety of foods?

L8: Importance of eating dairy to form strong bones?

L9: Mentioned that everyone needs 3 servings of dairy.

L10: Talked about risks of not having enough calcium.

L11: Did the presenter explain what a ‘serving’ is?

L12: Did the presenter give examples of ‘servings’?

L13: Presenter gave examples of different dairy options?

L14: We must ensure that we get 3 servings of dairy every day?

SCRIPT ADHERENCE - Laminates Discussed

The average script adherence was 76%

93%

86%

64%

79%

79%

79%

71%

71%

64%

64%

71%

79%

79%

79%

7%

14%

36%

21%

21%

21%

29%

29%

36%

36%

29%

21%

21%

21%

yes no

12

7%

21%

0%

14%

7%

0%

7%

0%

0%

0%

7%

7%

21%

7%

93%

79%

100%

86%

93%

100%

93%

100%

100%

100%

93%

93%

79%

93%

AUDIENCE ENGAGEMENT - Questions Asked

The audience engagement rate was 7%

THE QUESTIONS ASKED WERE:

I am a diabetic person, am I allowed to eat dairy?

Is it better to save than buy junk food?

What if the products make my stomach run?

What causes constipation in my stomach, after I have eaten?

Does milk cause worms?

Is it fine if I drink milk only and not drink water?

Is it possible to get a copy of the presentation?

How are we going to afford to drink milk 3 x a day?

Does butter contain dairy milk?

Is skimmed milk also good for the bones?

Can I drink milk directly from the cow?

Is Cremora good for consumption?

Can a person with hypertension consume dairy products?

What happens when the milk expires, can I still drink it?

When the milk is warmed up does it loose its proteins?

Can people with hypertension consume dairy products?

Are all companies processing milk in the same way?

L1

L2

L3

L4

L5

L6

L7

L8

L9

L10

L11

L12

L13

L14

yes no

AUDIENCE ENGAGEMENT

Most questions were asked around the following slides:

14

REACH - The number of people in the audience

Average reach of 24 people per education session

Thafamasi, KZN

Martjie Venter Hospital, Eastern Cape

Cradock Hospital, Eastern Cape

Sivananda Clinic, KZN

Joubert Park Clinic, Gauteng

Calcutta Clinic, Mpumalanga

AVERAGE

Kalafong Hospital, Gauteng

Tjakastad Clinic, Mpumalanga

Nhlazatshe 4 Clinic, Mpumalanga

Oukasie Clinic, North West

Kwamaphumlo Clinic (EDU-SHOW), KZN

Jeppe Clinic, Gauteng

Edenvale Hospital, Gauteng

Themba Hospital, Mpumalanga

3

7

14

18

20

21

24

30

32

41

50

80

90

153

200

15

REACH - Percentage of people who left the presentation

An average of 14% of the audience left the presentations, mainly because they went to

the doctor or moved to a different waiting area.

Calcutta Clinic, Mpumalanga

Cradock Hospital, Eastern Cape

Kalafong Hospital, Gauteng

Edenvale Hospital, Gauteng

Kwamaphumlo Clinic (EDU-SHOW), KZN

Sivananda Clinic, KZN

Themba Hospital, Mpumalanga

Joubert Park Clinic, Gauteng

AVERAGE

Nhlazatshe 4 Clinic, Mpumalanga

Thafamasi, KZN

Martjie Venter Hospital, Eastern Cape

Tjakastad Clinic, Mpumalanga

Oukasie Clinic, North West

Jeppe Clinic, Gauteng

38%

36%

27%

26%

24%

17%

17%

15%

14%

2%

0%

0%

0%

0%

0%

0

0

0

0

0

16

EXECUTION

The presenters were well spoken and very enthusiastic, however they had a challenging

audience at times

Did the presenter speak loud enough?

Did the audience understand the presenter’s language?

What was the presenter’s level of enthusiasm?

What was the audience’s level of interest?

What was the audience’s level of enthusiasm?

93%

100%

71%

43%

43%

7%

21%

29%

36%

7%

29%

7% 14%

very

good good fair poor

very

poor

REASONS FOR LOW ENTHUSIASM RATING

Consumers seemed bored with the presentation

Consumers were feeling unwell and struggled to focus

Some consumers were distracted by their children

17

ENGAGEMENT

The presenter’s level of engagement with the audience was outstanding, appropriate and highly

professional.

yes no unsure

Did the presenter

engage well with the audience?

100%

18

CONSUMER FEEDBACK

19

SAMPLE

The sample included 72 people.

Under 10

10-19

20-35

36-50

50 and over

7%

13%

43%

14%

22% 33%

67%

Female

Male

CONSUMER LEARNING

I learned something new about dairy today

17%

15%

13%

11%

11%

11%

7%

7%

7%

Strong bones

Different types of milk

Dairy can improve my health

Drink milk 3 x per day

Rich in calcium

Eat dairy 3 x per day

Cremora is not good for me

Pregnant people must eat 4 servings

Strong teeth

64% 29%

7%

We should eat dairy to maintain strong bones

Yes

No

Unsure

21

CONSUMER LEARNING

I learned something new about dairy today

There was nothing new for me to learn

I did not listen to the whole presentation

Dairy is too expensive so this presentation was of no interest to me

The presentation should have been in English I learned nothing moo today

Yes

No

Unsure

64% 29%

7%

22

SCRIPT RECALL - What are four examples of dairy?

61% of the people interviewed could name four examples of dairy correctly.

4 correct

3 correct

2 correct

1 correct

All incorrect

61% 23%

6%

10%

47%

32%

28%

22%

21%

13%

11%

11%

11%

8%

7%

6%

23

REASONS FOR EATING DAIRY PRODUCTS

People associate strong bones and body with consuming diary

Strong bones

Strong body and skin

Calcium

It is healthy

Protein

Makes me healthy again

Vitamins

Nutrients

Strong teeth

Energy gives us 'go'

Low fat

Reduces hypertension

Don’t mess with me

24

DAIRY COMPLEMENTS STARCH

63% of people believe that dairy makes the meal healthier

22%

13%

13%

11%

10%

10%

7%

6%

6%

3%

It tastes great

Grow strong / strong bones

I don't know

Add calcium

To add nutrients

To be healthy

Add protein

To add vitamins

Gives us 'go' / energy

Acts as 'spread'

Why should we add dairy to pap

and bread?

25

CONSUMPTION

Only one third of the people consume dairy daily

Daily

4-6 times a week

1-3 times a week

Less than once a week

Unsure

REASONS FOR CONSUMING DAIRY LESS OFTEN

I know that dairy is good for me but cheese is too expensive

I cannot afford milk every day

We are not in the habit of consuming dairy every day.

31%

28%

11%

7%

24%

15%

14%

51%

7%

3%

6%

4%

24%

25%

18%

4%

0%

3%

26%

CONSUMPTION

Many people don’t know how much dairy they consume.

1 serving per day

2 servings per day

3 servings per day

4 servings per day

5 servings per day

Many servings per day

Unsure

0

I should

eat (per

day)

I eat (when

I have

dairy)

26

27

SUMMARY AND RECOMMENDATION

IMPLEMENTATION DASHBOARD 2014

Specialised presentation Script adherence Duration

No planning errors With planning errors

High implementation rate 93%

Low defect rate 7%

Promotional material

Low implementation rate 58%

Low implementation rate 43%

High defect rate 57%

Implementation rate 78%

High defect rate 22%

Laminates shown 80%

Laminates discussed 76%

Average 26 minutes

Shortest 13 minutes

Longest 50 minutes

29

IMPLEMENTATION DASHBOARD 2014

Audience interest/enthusiasm Engagement levels

Very good / good 75%

Poor 25%

Size of audience % of audience who left

Average 14%

Highest 38%

Presenters - very

high 100%

Audience - very low 7%

Execution

Low average size 24

Highest (Themba Hospital) 200

Lowest (Thafamasi Clinic) 3

Very good / good 98%

30

IMPLEMENTATION DASHBOARD 2014

Dairy consumption Servings of dairy *

Moderate learning level 64%

People who learned

nothing new

29%

Learning level Script recall

1 serving 24%

2 servings 25%

3 servings 18%

4 servings + 7%

I dont’ know 26%

Main associations with dairy

Daily 31%

4-6 times a week 24%

Less than 4 times a week 39%

Could name 4

examples of dairy 61% Strong bones 47%

Strong body and skin 32%

* This refers to the numbers of

servings people have on the

occasions when they do eat dairy.

31

RECOMMENDATION

THE VENUE CHOICE IS CRITICAL

1

When people are seated in a dedicated waiting

room the attention levels are higher compared to

when people are standing or are seated in

corridors.

Not all venues worked well for this presentation.

For example, at times the clinic staff stopped and

then re-started the presentation.

At times there were very few people present.

Choose the location carefully based on seating

arrangement and venue layout.

Avoid clinics with high traffic and resulting high

noise levels.

Choose a venue with an appropriate number of

patients.

1

2

3

32

GOOD vs BAD VENUES

Good

Bad

Bad Good

Bad Bad

Effective space for presentation Effective space for presentation A corridor is not a good place for a

presentation

Only three people in the audience The TV is distracting even when muted Only three people in the audience

33

RECOMMENDATION

THE PRESENTATION (content and process)

2

At times the presenter rushed through the slides,

not giving the audience time to digest the learning

or ask questions.

Some presenters opted not to show all material

which might be against the given task, however

the presenter might have ‘read’ the audience and

focused on what he/she thought is most important.

The audience engagement level was extremely

low.

Some presenters need to slow down their

presentations. The schedule is sometimes too tight

and presenters do not have a lot of time to move

from one venue to the next.

Some flexibility in the script might be an effective

alternative to rushing through all the slides.

Slow down the presentation and ask the audience

questions to motivate engagement.

4

5

6

34

RECOMMENDATION

IMPLEMENTING PROMOTIONAL MATERIAL

3

The level of implementation was moderate.

Ask the presenters to take a picture of the

setup to monitor the implementation

continually.

7

Monitor the implementation, success and

the programme all times.

8

KEEP OBSERVING AND IMPROVING

4

The evaluations track the effectiveness and the

success of the campaign. On-going evaluations will

monitor the progress and establish success trend

lines.

The evaluations also motivate the promotion

company to perform at all times.

35

SUMMARY OF RECOMMENDATIONS

Choose the location

carefully

Avoid noisy, busy clinics

Choose a venue with

enough patients

Present more slowly, allow

more time between

presentations

1

2

3

4

Make the script more flexible

to avoid rushing

Ask audience questions to

motivate participation

Take pictures of the setup

Keep observing, monitoring,

evaluating and improving

5

6

7

8

36

THANK YOU

ABOUT MQ MARKET INTELLIGENCE

WHO WE ARE OUR APPROACH RESEARCH SERVICES

MQ Market Intelligence is a full

service market research company

specialising in customised

quantitative and qualitative

consumer and business-to-

business research.

Our approach emphasises one-

on-one personal interactions to

ensure a full understanding of

your research requirements. At

the same time we are at the

cutting edge of new research

techniques and analysis tools

ensuring our clients the most

appropriate and effective project

execution.

Helping our clients to understand

what their customers mean, not

just what they say, is the

heartbeat of what we do.

Advertising evaluation

Consumer attitude and usage

studies

Customer satisfaction studies

Desktop research

Feasibility studies

Multi-client customised consumer

research

Mystery evaluations

Product placements and

acceptance tests

Tracking studies