What works? Policy design without theory is useless

Against a rationalist top down approach to policy making, the evidence-informed policy and practice has rapidly evolved in the last two decades.

In this line of research, a new book What Works Now? Evidence-informed Policy and Practice has been edited by Annette Boaz, Huw Davies, Alec Fraser and Sandra Nutley.  It offers not only a synthesis of the role of evidence in policy making but also an analysis of its use in recent economic models and practices in the UK, Australia, New Zealand, Scandinavia, Canada and the United States. In addition to the diversity of policy and practice settings where evidence is sought and gets applied, the book considers policy examples related to healthcare, social care, criminal justice, education, environment and international development.. At the core of the argument regarding the actual relevance of ‘know-about’, ‘know-what works’, ‘know-how’, ‘know-who’ and ‘know-why’ is the belief that evidence matters.

Considering this policy scenario, the relevant question at stake is  what are the implications of the new policy design practices that mainly rely on the belief that evidence matters?

What is important to note is that behind the belief that evidence matters is a deep transformation of the public policy approach  towards a more experimental and empirical one.

At this respect, in the UK, the Nudge Unit leader David Halpern recently suggested the conceptualization of experimental government  in order to characterize the new approach to policy making based on evidence. The relevance of the potential outcomes of  systematic testing is clear in Halpern’s words:

Governments, public bodies and businesses regularly make changes to what they do. Sometimes these changes are very extensive, such as when welfare systems are reformed, school curricula are overhauled, or professional guidelines are changed. No doubt those behind the changes think they are for the best. But without systematic testing, this is often little more than an educated guest. To me, this preparedness to make a change affecting millions of people, without testing it is potentially far more unacceptable than the alternative of running trials that affect a small number of people before imposing the change to everyone. 

At the heart of his proposal about “what works best” in public policy is the use of evidence as a regular practice to select the measures that actually operate in a more efficient  way.  Moreover, no  ethical considerations about the efficient methods and goals in policy making are added to his explanation.

Taking into account the methodologies that support some policy practices that favour inductive reasoning and randomized control trials of impact evaluation (RCTs), there is a controversy around the utilization of these attempts to build experimental programmes or policy intervention. For instance, Deaton and Cartwright (2016) pointed out that there are misunderstandings around what the RCTs can really do. For them, the inductive techniques used in research do not guarantee that the relevant causal factors are taken into account across sample groups in any specified RCT. Therefore, the results of the inference pocess might be wrong. Indeed, the outcomes of RCTs can be challenged ex post, after examining the composition of the control group and the factors considered in the experimental setting. Moreover, Deaton and Cartwright also rejected the transportation of the outcoems of RCTs to other contexts since the relations of causality between variables is always context-dependent.

As the decision-making policy process in the real world relies on institutional factors that may be different elsewhere, the methodology based on RCTs does not provide a credible basis for policy making. In short, the outcomes of inductive investigation can never be completely transported across time and space.

Moreover, economists Steven D. Levitt and John A. List (2007) highlighted that human behaviour in RCTs can be affected by the selection of the individuals, the evaluation of their actions by others, and ethical issues. Then, the findings in a laboratory setting may overestimate or underestimate the effectiveness of policy interventions within real life interactions. In other words, if a policy intervention “works” and makes people better off in a laboratory, there is no guarantee that this intervention may actually do so in the real-world.

In fact, the methodology of RCTs runs the risk of considering worthless casual relationships as relevant causalities in the attempt to develop policy recommendations. In short, the use of the outcomes of RCTs as normative orientations for policy making should be put in question.

“What works” in the “sterile” environment of a laboratory does not necessarily work in a real-world where social interactions and the dynamics of institutions are overwhelmed by power relations. Therefore, ethical considerations should be considered in any attemp to  build policy proposals.

Indeed, the transformation of the economic policy approach has evidently been a remarkable one. It is worth recalling the words of Lars Syll about the current sad state of economics as a science,

A science that doesn’t self-reflect and asks important methodological and science-theoretical questions about the own activity, is a science in dire straits. The main reason why mainstream economics has increasingly become more and more useless as a public policy instrument is to be found in its perverted view on the value of methodology.

 

 

References
Boaz, A, Davies, H., Fraser, A and Nutley, S. (eds) What Works Now? Evidence-informed Policy and Practice.. Policy Press. 2019.,
Deaton, A. and Cartwright, N. (2016). Understanding and misunderstanding randomized controlled trials. NBER Working Paper No. 22595.
Halpern, D. (2015). Inside the Nudge Unit: How Small Changes Can Make a Big Difference. London: WH Allen.
Levitt, S. D. and List, J. A. (2007). What do laboratory experiments measuring social preferences reveal about the real world? Journal of Economic Perspectives, 21 (2): 153–174.
Madi, M.A.C (2019). The Dark side of Nudges. London: Routledge.
Sill, L. (2019=. Economics becomes more precise and rigorous — and totally useless
April 4. https://rwer.wordpress.com/2019/04/04/economics-becomes-more-precise-and-rigorous-and-totally-useless/