A Step-by-Step Guide:Conducting Experiments in Public Policy

CONTENTS

How to Use this Guide iii

Overview: 5-Step Approach to Conducting Experiments in Public Policy 1

5-Step Approach

Step 1: Analyse Problem 2

Step 2: Problem Statement 3

Step 3: Design Interventions 4

Step 4: Select a Testing Approach 6

Step 5: Evaluate Interventions 9

Annexes

Annex A: Problem Definition & Theory of Change 11

Annex B: Cost Benefit Analysis 12

Annex C: Evaluation Methods 13

Annex D: Randomisation on Excel 14

Annex E: Calculating Sample Size 15

Annex F: t-Test on Excel and Interpreting P-values 16

Annex G: Null Hypothesis and Alternative Hypothesis 18

Additional Resources 19

References 20

ii

The Civil Service College, Singapore, firstpublished “Evidence-based Policymaking inSingapore: A Policymaker’s Toolkit” in 2016. Itprovided an overview of tools commonly used inevidence-based policymaking, and how they fitinto the policy development process.

This step-by-step guide is a deeper dive into the 3stages within the policy development andevaluation cycle, which are Problem Definition,Solution Development, and Analysis andRecommendation (see Figure 1).

It outlines 5 key steps, which will help you workthrough these stages more effectively. Theyinclude understanding the root of the problemthat you are trying to solve, designing potentialsolutions, and testing whether or not theseproposed solutions work in solving the problem.

HOW TO USE THIS GUIDE

Throughout the guide, a project collaborationbetween CSC and the Land Transport Authority,Singapore called “Project Tap-Out”, has beenused to illustrate the 5 steps (see Figure 2). Thisproject aimed to nudge commuter habits withBehavioural Insights. The published article onthis project and its findings can be found athttps://www.csc.gov.sg.

Lastly, we encourage you to explore theresources listed in the final section to elevateyour ability to apply the steps outlined in thisguide. These resources have been put togetherbased on our experience of commonly-askedquestions and challenges, in our discussions withpublic officers.

We hope you will find this guide a usefulcomplementary resource to your work!

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESO

VER

VIEW

CO

NTEN

TS

iii

ANALYSE PROBLEM

Start your project with a well-definedresearch question.

1

PROBLEM STATEMENT2

DESIGN INTERVENTIONS3

SELECT A TESTING APPROACH4

EVALUATE INTERVENTIONS5

Generate possible interventions andselect the intervention(s) for yourproject.

Find out the who, what and why of theproblem.

Assess the intervention’s process,outcome and impact. To guide scalingup efforts, continue to tracking andmonitor.

Decide on your testing approach toknow what would work and would notwork.

Note: It is not always a linear step-by-stepapproach. You may have to revisit some of thesteps at times.

5-STEP APPROACH TO CONDUCTING EXPERIMENTS IN PUBLIC POLICY

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESO

VER

VIEW

Source: Adapted from “Policy Development in Practice: AnOverview of the Policy Process” (2007), Civil Service College

This guide brings you through the process involvedin the first 3 steps of the policy development andevaluation cycle (see Figure 1).

CO

NTEN

TS

1

Figure 1: Policy Developmentand Evaluation Cycle

Figure 2: 5-Step Approach to Conducting Experiments in Public Policy

o Administrativedata

o Surveys

QUANTITATIVE DATA

o Interviews

o Focus Group Discussion(FGD)

o Pilot or test prototypeswith target group toobtain their feedback

o Ethnographic research:observe target group intheir own environmentand interview themduring the process.E.g. create a journeymap that outlines allthe touchpoints of thetarget group in theprocess.

QUALITATIVE DATA

Problem Analysis for Project Tap-Out:Nudging Commuter Habits withBehavioural Insights

Problem: Given that Monthly ConcessionPass (MCP) holders pay a flat fee forunlimited bus rides within the month, theydo not have any incentive to tap out whenexiting a public bus. This affects the accuracyof data that is used for bus route planningand bus loading computations.

Gather Quantitative and Qualitative Data

• Who? (Target Group): This problem wasidentified among the MCP holders,especially Polytechnic students who hadthe highest non-tap-out rates amongMCP holders.

• What is the Baseline Behaviour?: Theaverage tap-out rate of Polytechnic MCPHolders was 41%. In contrast, theaverage tap-out rate for all MCP holderswas 47.4%.

• Why? (Reason for Baseline Behaviour):Based on insights from the FGDs, manypolytechnic students did not tap out dueto peer influence (“my friends told methere was no need to do so.”).

When students at the FGDs were toldthat tapping out helps the governmentplan bus services better, they seemed torespond positively. This provided aninsight for the design of the intervention.

Box 1STEP 1: ANALYSE PROBLEM

When faced with a policy or intervention review,start by analysing the problem with the followingquestions:

a) Who is the target group? For example, do theybelong to a certain demographic orsocioeconomic group?

b) What has been the baseline behaviour (i.e.current behaviour)? This will help you identifythe gap between the current and desiredbehaviour, which is important in evaluating theeffectiveness of a new intervention.

c) Why is the target group not showing the desiredbehavioural outcome? This information willguide the design of interventions that can helpaddress the barriers to desired behaviour.

To answer these questions, you will need to gatherquantitative and qualitative data, which can surfacedeeper insights to design suitable interventions.

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

2

Box 2

The lack of clarity may result in a myriad of

interpretations. For instance, “How is

Singapore’s labour market doing?” is framed

too broadly and does not pinpoint the exact

problem or aspect of the labour market. Is the

question interested in employment rates,

incentives or discrimination practices?

C LEAR

I MPACTFUL

A NSWERABLE

The potential impact of the results obtained

is crucial. There needs to be significant

impact that can inform policy design and

approaches in order for the study to be seen

as relevant by key stakeholders.

Any research question posed should be onethat can be answered in a scientific manner.Are there quantitative and/or qualitative datathat could be used as evidence to support orfalsify what we believe is true?

A Specific Problem StatementHow can we use Behavioural Insights to increase the tap-out rates of Monthly

Concession Pass (MCP) Holders on public buses for polytechnic students?

IMPACTFUL

MCP holders are subsidised by full-farepaying commuters. If MCP holders donot tap out, they appear to betravelling longer distances and enjoy alarger amount of fare subsidy. Hence, ahigher tap-out rate ensures that theaccurate amount of fare subsidy goesto the MCP holders.

ANSWERABLE

The effectiveness of the possibleinterventions can be measuredobjectively and evaluated by trackingthe tap-out rates of Polytechnic MCPholders.

CLEARThe specific problem (tap-out rates)and target group (Polytechnic MCPHolders) are identified.

Source: Evidence-based Policymaking in Singapore: APolicymaker’s Toolkit”, Civil Service College 2016

STEP 2: PROBLEM STATEMENT

Start with a Specific Question

Getting insight to what works depends heavily onhow well-defined your problem statement is, rightfrom the start.

Check if the problem statement is:

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

3

Note: LTA was interested in using BI as analternative approach to traditional policyinstruments like heavy penalties orregulations.

Print Annex A to guide you through Steps 1 and 2.

Applying the EAST Framework

Small financial incentives (gain and lotterygroup) were used to motivate the desiredbehaviour.

ATTRACTIVE

The gain, loss and lottery groups receivedfortnightly emails which included updateson their tap-out behaviour as well as theincentives/lottery outcomes. The emailnotifications sent were timed this way, sothat participants were frequently remindedof their progress and incentives.

TIMELY© Behavioural Insights Ltd

EASY

TIMELY

SOCIAL

ATTRACTIVE

Source: UK Behavioural Insights Team 2014

Box 3

Identifying Possible Interventions for ProjectTap-Out: Using Behavioural Insights toIncrease the Tap-Out Rates of PolytechnicMonthly Concession Pass (MCP) Holders onPublic Buses

This project designed the followinginterventions based on data collected in Step 1.

1) Information only group: Students were toldwhy tapping out is important.

2) Gain group: Students were told the sameinformation as the information only group,and were paid a bonus for tapping out.

3) Loss group: Students were told the sameinformation as the information only group,and were charged a penalty for not tappingout.

4) Lottery group: Students were told the sameinformation as the information only group,and were entered into a lottery if theytapped out.

(a) Identify Possible Interventions

Now that you are clear about the root of theproblem, generate possible interventions (e.g.nudges) that will address the barriers to desiredbehaviour identified in Step 1. Going through pastresearch on similar projects is a good starting pointto gather ideas.

The EAST framework developed by the UKBehavioural Insights Team (BIT) provides the keyprinciples to designing effective nudges. They needto be:

• Easy (simple and causing least friction);

• Attractive (attracting attention);

• Social (social norms and team spirit);

• Timely (delivered at the right time by the rightperson).

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

4

STEP 3: DESIGN INTERVENTIONS

Depending on the context, nudges need not bedesigned with all the elements of the EASTframework. Combine this with other useful toolssuch as Design Thinking to come up withinterventions to test and try out.

Refer to Annex A and check if your intervention isaligned to your Theory of Change.

Prioritising Interventions to Test in ProjectTap-Out

Students at the Focus Group Discussionswere shown examples of possibleinformation nudges as seen below. Theyseemed to respond positively to tapping outwhen given a simple reason to do so.However, when the students wereintroduced to a more complex reason, themessage was lost on them and did not seemuseful as a nudge.

This insight was useful in refining theintervention for the project, whicheventually used the simple message as oneof the interventions.

Box 4

Examples of Information Messages• Simple Reason: “It is important that you

tap out … This is because by tapping out,you give the government more accurateinformation about your bus journeys andhow crowded buses truly are. With thisinformation, the government can do abetter job of improving bus servicesacross Singapore.”

• Complex Reason: “I already enjoy asubsidy when I travel on a monthlyconcession pass. If I do not tap outwhenever I take the bus, other publictransport users / government may haveto further cross-subsidise my bus fares.This is unfair to other public transportusers.”

(b) Prioritise the Interventions to Test

Given limited budget and resources, you will not beable to test all the possible interventions. Prioritisethe interventions by considering the followingquestions:

Based on the data gathered from FGDs andliterature reviews, which intervention is mostlikely to work?

Will the project generate net benefits? Proceedwith testing only if your Cost Benefit Analysis(see Annex B) reveals that the best case scenariowould result in a positive net impact. You willhave to make projections to estimate the futurebenefits and costs.

What is the likelihood of getting approval for theintervention from your stakeholders?

What kind of support do you need to carry outthe experiment?

(c) Refine Intervention Before Testing

Before testing, present the first iteration of theintervention(s) to a small group to elicit feedback.This group should be representative of thedemographics of the target group (i.e. at least onefrom each demographic group).

Based on the feedback collected, refine theintervention(s) to maximise the desired impact (seeBox 4).

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

5

STEP 3: DESIGN INTERVENTIONS

Methodology

Figure 3: Common Evaluation Methods

Qu

asi-

Exp

eri

me

nts

Description

Source: Abdul Latif Jameel Poverty Action Lab 2016

STEP 4: SELECT A TESTING APPROACH

(a) Select a Testing Approach

Testing your intervention is important todetermine whether it works. Figure 3summarises the common evaluation methods.

Out of the evaluation methods, RandomisedControlled Trials (RCTs) are considered thegolden standard for evaluation because theyenable a robust and clean evaluation of howeffective a new intervention is. However, RCTscan sometimes be misperceived to beunethical and costly.

Fact #1: RCTs can be ethical.

If you have plans to run a pilot or introduce apolicy in phases, why not take an opportunityto run an RCT? Groups that are scheduled toreceive the intervention at a later stage can bethe control group in the RCT. Running an RCTbefore full implementation is crucial becausewe can never be certain if the intervention willbe ineffective or harmful. As long as thecontrol group is not made worse off relative tothe status quo, RCTs can be ethical.

Fact #2: RCTs need not be expensive.

In cases where complete randomisation iscostly and may take a long time to administer,you could explore the following:

• Carry out experiments on interventionswhere outcome data is already beingcollected systematically (e.g. administrativeand survey data). The cost ofadministration and running the trial can belowered.

• Consider randomising at larger units ofmeasurements (e.g. classes in schools,housing blocks) instead of randomising atthe individual level.

Randomised Controlled Trial (RCT)

Assigns the intervention randomly toindividuals (treatment group), andwithhold the intervention from therest (control group). Both groupsshould be alike statistically in theircharacteristics. Read Annex D to findout how you can randomise on Excel.

Regression Discontinuity Design

Makes use of the eligibility criteria inpolicies to compare those whoreceived the intervention (i.e. meetthe criteria) with those who falloutside the cut-off.

Multiple Linear Regression

Compares impact on the treatmentgroup and control group by adjustingfor observable characteristics (e.g.income, gender, age) that might causedifferences in outcomes between bothgroups.

Differences in Differences

Measures the effect of theintervention by finding the differencebetween the change in the outcome ofthe treatment group and the changein the outcome of the control group.

Pre-Post Test(Least reliable approach)

Measures the change in outcomebefore and after the intervention;there is no concurrent control group.Use this method if the intervention isthe only factor influencing theoutcome, but normalisation isnecessary to ensure that the resultscan be easily compared.

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

6

Read Annex C for more information on evaluationmethods.

Box 5

Selecting a Testing Approach for ProjectTap-Out

An RCT was conducted for this project. Thedata collection was facilitated by the factthat each student had an individual travelcard that could be easily tracked in abackend system.

Attrition was not likely as travelling toschool daily by bus was not likely tochange, and students did not need to putin any extra effort to have their datacaptured in the system.

There was contamination across thetreatment groups, but this was containedby sending students personal emailreminders.

The eventual RCT design included 1 controlgroup and 4 treatment arms as mentionedin Box 3 earlier. The control group providedthe baseline behaviour to compare theeffectiveness of interventions.

STEP 4: SELECT A TESTING APPROACH

When Not to Use RCTs?

RCTS would not work well in these situations:

× When a policy has already been rolled out to aselected group, which could potentially result inself-selection or sampling biases.

× Where there are likely to be interactions betweencontrol and treatment groups, which could result incontamination of the experiment.

× When it is not possible to ensure minimal attritionand good compliance of the treatment.

Under such circumstances, the next best alternativewould be to explore the use of quasi-experiments (seeFigure 3). If you are unable to use more sophisticatedquasi-experimental techniques like regressiondiscontinuity design, we encourage you to conduct asimple pre-post test to determine the effectiveness ofyour intervention.

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

7

How Large Should the Sample Size Be?

A large sample size increases the chances ofobtaining a statistically significant result. ReadAnnex E to find out how to calculate the minimumsample size required for your intervention.

When Is a Small Sample Size Acceptable?

If there is more homogeneity in the sample (e.g.similar demographics), you would not need such alarge sample size given that there would be asmaller variance between the control andtreatment group.

(b) Key Elements of Running an Evaluation

Now that you have selected an evaluation method,these are the key elements to think about whenrunning an evaluation.

True Randomisation or Stratification?

True randomisation involves randomly assigningindividuals to the treatment or control group.However, if the sample size is not large enough, thiscould result in certain groups showing a higherlikelihood of reacting to the intervention (e.g. atreatment group in a clinical trial has a greaterproportion of unhealthy individuals compared to thecontrol group).

Stratified randomisation is one way to get around thisproblem. This method involves these steps:

(i) Construct categories (i.e. strata) according to keyfactors that could affect the outcome. Forexample, in the context of encouraging companiesto increase their donation amount to charities,sort the companies into two strata (i.e. frequentand infrequent donors).

(ii) In each stratum, randomly assign the participantsto the treatment or control group.

The purpose is to ensure that the compositionbetween the control and treatment group is not vastlydifferent. This approach is especially important if yoursample size is small given that there is a higherlikelihood of imbalances among the groups.

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

8

STEP 4: SELECT A TESTING APPROACH

Every experiment carries the hope of havingstatistical significant results. This means thatthe intervention tested brings about a realchange to the treatment group compared to thecontrol group, and this difference in outcomedid not occur by chance.

In statistics, null and alternative hypotheses(see Annex G), and p-value are used to check ifthe results occurred by chance. The generalassumptions are:

• Null hypothesis (Ho): the results are due tochance

• Alternative hypothesis (Ha): the results arenot due to chance.

• P-value: the probability that the nullhypothesis is true.

To find out more about conducting a t-test andinterpreting p-values, refer to Annex E.

Box 6

Assessing the Results for Project Tap-Out

Key findings from the RCT shed light on whatworked and other related aspects such asframing information effectively and habitformation:

• The increase in tap-out rates for the lossgroup was statistically significant whencompared to the information group, but… providing information alone alreadyresulted in a significant increase in tap-outs compared to the control group.

• The full effect of behaviours werereached and sustained within the firsttwo weeks of the experiment.

Post-RCT FGD (Qualitative Insights)

• It was revealed that the action of tappingout was not too onerous and for somestudents, the information nudge wassufficient to motivate them to tap out.However, the information has to besimple to understand, credible andfocused on the tangible benefits (e.g.timelier arrivals of buses).

Figure 4: Statistical Significance

P-value 0.05 or 0.1

benchmark

<Result is statistically significant if:

Read the “Evidence-based Policymaking inSingapore: A Policymaker’s Toolkit” for more ondata analytics tools.

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

9

(a) Post Experiment: Assess Interventions

After the experiment, evaluate whether the intervention was implemented as intended (i.e. ProcessEvaluation) and whether it achieved the goals (i.e. Outcome and Impact Evaluation).

(i) Process Evaluation: Using qualitative and/or quantitative methods, assess how well the programmewas implemented. This includes the quality of operations (e.g. service delivery, adherence to statedplans), intervention’s deliverables (e.g. programme reach) and acceptability to stakeholders (e.g.participants’ sentiments and responsiveness).

(ii) Outcome & Impact Evaluation: Conduct a statistical analysis to determine if the result is statisticallysignificant (see Figure 4). For example, run a t-test on Excel if you have a small sample size (see AnnexF). If the result is not statistically significant, there are other ways to deepen insights to what workedand what did not, such as gathering qualitative data (e.g. FGDs) on the experiment.

STEP 5: EVALUATE INTERVENTIONS

Tracking and Monitoring Project Tap-OutPost-Experiment

To determine if the change in behaviourwould be sustained, participants’ tap-outrates were tracked for 8 weeks after theexperiment.

• It was found that all participantstapped out more than pre-experimentrates, albeit there was some decline.

• However, by week 6 after theexperiment ended, the tap-out ratefrom the information group was higherthan the tap-out rates from themonetary intervention groups. Thissuggests habit formation was more‘sticky’ and sustained in theinformation only group.

These insights were useful inunderstanding how to use informationmore effectively for similar policychallenges.

(b) Post Experiment: Scale Up or Discontinue?

• Continue tracking and monitoring the results evenafter the experiment. They provide invaluableinsight to habit formation and sustainability. Theremight be changes in outcomes in the longer term.Be prepared to continue iterating and refining theintervention.

• Determine whether the results from evaluation(e.g. impact evaluation and cost benefit analysis),justify scaling up the intervention.

• Before scaling up, check if the intervention is stillaligned to the initial Theory of Change (i.e. ContextEvaluation). Changes in external environmentwould require you to review the Theory of Change(see Annex A) and refine the interventionaccordingly.

Box 7

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

10

STEP 5: EVALUATE INTERVENTIONS

ANNEX A: PROBLEM DEFINITION & THEORY OF CHANGE

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

11

Print this template to guide you through the problem definition and your theory of change.

Source: Campbell and Brown 2003

Figure 5: Cost Benefit Analysis

ANNEX B: COST BENEFIT ANALYSIS

What Is Cost Benefit Analysis (CBA)?

It is a useful decision-making tool to estimatethe net value of a project. Specifically, itassesses the costs and benefits of undertakingthe project compared to not undertaking it(see Figure 5). If the benefits generated by theproject exceed its opportunity costs, theproject is worth pursuing.

CBA also includes going beyond the monetarybenefits and costs, and should take intoaccount non-quantifiable benefits and coststhat are not reflected in market prices (e.g.benefits of a recreational park).

When Can CBA Be Used?

In this guide, we gave an example of using CBAbefore the project (see Step 3). However, CBAcan also be used after a project to determinethe project’s net impact on society. Conductinga CBA after the project has been implementedenables you to use actual data to evaluate theproject’s progress. This would be useful inrefining the intervention to maximise thebenefits.

Source: Evidence-based Policymaking in Singapore: APolicymaker’s Toolkit”, Civil Service College 2016

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

12

Source: Abdul Latif Jameel Poverty Action Lab 2016

ANNEX C: EVALUATION METHODS

Randomised Controlled Trial (RCT)

Assigns the intervention randomly toindividuals (treatment group), andwithhold the intervention from the rest(control group). Both groups should bealike statistically in their characteristics;the intervention should be the onlydifference.

Randomisation is possible and thereshould not be contamination across thecontrol and treatment groups. Theoutcome(s) should also be easilymeasured.

Regression Discontinuity Design

Makes use of the eligibility criteria inpolicies to compare those who receivedthe intervention (i.e. meet the criteria)with those who fall outside the cut-off.

Individuals who meet the eligibility criteriaand those who fall outside the cut-offshould be statistically identical. The cut-offpoint should not be manipulated to ensurethat certain individuals receive theintervention.

Multiple Linear Regression

Compares impact on those who receivedthe intervention and those who did notreceive the intervention by adjusting forobservable characteristics (e.g. income,gender, age), which might causedifferences in outcomes between bothgroups.

Characteristics that were excluded fromthe experiment do not affect the outcomeor do not differ between both groups.

Differences in Differences

Measures the effect of the intervention byfinding the difference between the changein the outcome of the treatment group andthe change in the outcome of the controlgroup.

Without the intervention, the outcome forparticipants and non-participants wouldexperience the same identical trajectoriesduring the study period.

Pre-Post Test(Least reliable approach)

Measures the change in outcome beforeand after the intervention, where there isno concurrent control group.

Use this method if the intervention is theonly factor influencing the outcome, butnormalisation is necessary to ensure thatthe results can be easily compared.

DescriptionMethodology When can it be used?

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

13

These are common evaluation methods to test whether the intervention worked.

Step 1: Enter the fulllist of participants inColumn A.

Step 2: Type =RAND() in cell B2.Copy this cell and paste thisdown Column B till the end ofthe list of participants.

Step 4: Type =RANK(B2,B$2:B$16) in cell C2.Copy this cell and paste this down Column C tillthe end of the list of participants.

Step 5: Click on Cell C1. Next, click the DATA taband click on SORT FROM A to Z.

Step 3: Select all the values in Column B and rightclick on these cells. Select COPY. Next, right clickon the highlighted cells again and select PASTEVALUES.

Step 6: Assuming you have 1 control group and 1treatment group, allocate an equal number ofparticipants to the control and treatment groupas shown in Column D.

ANNEX D: RANDOMISATION ON EXCEL

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

14

𝑛 =𝑧 + 𝑏 2 𝑥 2𝑠2

𝑑2

• z represents the confidence interval, which indicates the likelihood of drawing a false positive conclusion.A false positive conclusion is when the experiment results happened given that the null hypothesis is true.Conventionally, the standard confidence level is 95% (i.e. z = 1.96).

• b represents the beta, which is the likelihood of a false negative conclusion. A false negative conclusionarises when the null hypothesis is not rejected when it is false. The conventional value for b is 0.2, whichmeans that there is a 20% chance of a false negative conclusion.

• s represents the standard deviation, which is a measure of variability in the data set. This value isdetermined based on the expected variance.

• d is the expected effect size you would like to see between the 2 groups.

Source: Behavioural Insights Playbook for Channel Shift, Public Sector Service Delivery Council 2017

ANNEX E: CALCULATING SAMPLE SIZE

Use the formula below to calculate the minimum sample size (n) required for your intervention.

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

15

Step 1: Key in the data for your controland treatment group in Column A andB. Click FILE and OPTIONS. Click theAdd-Ins category. Under Manage box,select Excel Add-Ins and click Go.

Step 2: In the Add-ins box, check Analysis Toolpak and click OK.

Step 3: Click the DATA tab and then DATA ANALYSIS.

Continue to the next page for Steps 5 – 9.

Step 4: Select the type of t-test you need. Thecommon t-test used is the ‘two sample assumingunequal variance’.

ANNEX F: T-TEST ON EXCEL AND INTERPRETING P-VALUES

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

16

Step 7: Check the ‘Label’ box. Inmost studies, we can leave thealpha value (significance level) as0.05

Step 5: Select the cells from the start of your datapoint to the end of the data point in the controlgroup.

Step 6: Select the cells from the start of your datapoint to the end of the data point in the treatmentgroup.

Step 8: Select the OUTPUTRANGE option and click on thecell (e.g. D2) you would like toinput the t-test results. Next, clickOK.

Step 9: Your t-test result is now shown. For a morerigorous evaluation of the results, look at the two-tail values.

• If the t stat is greater than the t critical (twotail), we can reject the null hypothesis. You canignore the negative sign when comparing thetwo t-values.

• The results are statistically significant if the p-value is lower than a benchmark (usually 0.05or 0.1), which means that the probability of thenull hypothesis being true is less than 5% or10%. This is a small enough probability that wecan reject the null hypothesis and conclude thatthe results are not due to chance.

In this case, we cannot reject the null hypothesisgiven that the t-stat value (1.03) is less than tcritical value (2.04).

Note: You might be interested in using a one-tailedtest but consider the effect of not knowing theother direction of the effect. Only use a one-tailedtest if there is no implication or relevance ofknowing the other direction of the effect.

ANNEX F: T-TEST ON EXCEL AND INTERPRETING P-VALUES

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

17

To run a statistical analysis like a t-test, you need to determine the null and alternative hypothesis of yourexperiment.

ANNEX G: NULL HYPOTHESIS AND ALTERNATIVE HYPOTHESIS

Null Hypothesis (H0) Alternative Hypothesis (Ha)

Description The null hypothesis statement statesthe outcome/fact that you are trying todisprove in your experiment

The alternative hypothesis states the outcomeyou are trying to prove in the experiment.There are 2 types of alternative hypotheses:

(1) One-sided: A hypothesis that is used todetermine if the outcome differs from thehypothesised value in a specific direction.

(2) Two-sided: A hypothesis that is used todetermine if the outcome is either greateror less than the value under H0. (i.e.direction does not matter).

Example The average waiting time for publicbuses in Buona Vista is 15 minutes.

One-sided: With the introduction of bus lanes,the average waiting time for public buses inBuona Vista is shorter than 15 minutes.

Two-sided: With the introduction of bus lanes,the average waiting time for public buses inBuona Vista is shorter or longer than 15minutes.

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

18

CSC’s WORKSHOPS

• A Public Officer’s Toolkit for Designing and Evaluating Policies and Programmes (PDE10)

• Introduction to Behavioural Insights (EPD10)

• Cost Benefit Analysis for Beginners (CBAFC)

• Cost Benefit Analysis for Practitioners (CBASC)

• Cost-Effectiveness Analysis for Beginners (PKV10)

• STATA for Public Policy (ST101)

RESOURCES ON DEMAND

Publications

• Ethos Issue 17 (Behavioural Insights)

• Ethos Issue 12 (Randomised Controlled Trials in Policymaking)

Other Guides/Toolkits

• Evidence-based Policymaking in Singapore: A Policymaker’s Toolkit

• Public Service Innovation Framework Process Guide (Download the guide from the Workplace Group “[WOG] Public Service Innovation”)

COMMUNITY

ADDITIONAL RESOURCES

• Want to share your agency’s work or hear from others? Click here to join the Behavioural Insights & Design Thinking Community of Practice supported by CSC.

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESC

ON

TENTS

OV

ERV

IEW

19

Abdul Latif Jameel Poverty Action Lab. “Impact Evaluation Methods.” 2016. Accessed April 23, 2018. https://www.povertyactionlab.org/sites/default/files/resources/2016.08.31-Impact-Evaluation-Methods.pdf

Behavioural Insights Team. “EAST: Four Simple Ways to Apply Behavioural Insights.” 2014. Accessed April 19, 2018. http://www.behaviouralinsights.co.uk/wp-content/uploads/2015/07/BIT-Publication-EAST_FA_WEB.pdf

Behavioural Insights Team. “Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials.” 2013. Accessed April 19, 2018. http://38r8om2xjhhl25mw24492dir.wpengine.netdna-cdn.com/wp-content/uploads/2015/07/TLA-1906126.pdf

Campbell, H., and R. Brown. Benefit-Cost Analysis Financial and Economic Appraisal Using Spreadsheets. UK: Cambridge University Press, 2003.

Civil Service College. “Evidence-based Policymaking in Singapore: A Policymaker’s Toolkit” 2016. Accessed October 16, 2018. https://www.csc.gov.sg/resources/book-evidence-based-policymaking-in-singapore-a-policymaker-s-toolkit

The Public Sector Service Delivery Council. “Behavioural Insights Playbook for Channel Shift.” 2017. Accessed April 16, 2018. https://iccs-isac.org/assets/uploads/publications/PSSDC-Playbook-for-Channel-Shift-FINAL-2017-03-21.pdf

REFERENCES

20

STEP 1

STEP 2

STEP 3

STEP 4

STEP 5

AN

NEX

ESO

VER

VIEW

CO

NTEN

TS

Civil Service College, Singapore31 North Buona Vista Road Singapore 275983

www.cscollege.gov.sgwww.facebook.com/CivilServiceCollegeSingapore

© 2019 Civil Service College, Singapore. All rights reserved. No part of this paper may be reproduced, modified, stored in a retrieval system, or transmitted in any form by any means, electronic, mechanical, photocopying, recording or otherwise,

without the prior written permission of the Civil Service College, Singapore.