Entrepreneurship Policy Evaluation and the Role of Randomised Control Trials

What’s the issue?

Is entrepreneurship policy having its intended effects? Do the benefits of specific programmes justify their costs? Could these benefits be achieved more cost-effectively through alternative approaches? Evaluation provides the key to answering these questions, by determining the relevance, efficiency and effectiveness of a policy or programme in relation to its objectives. However, reliable impact evaluation is rare in the field of entrepreneurship policy. This is due in part to a lack of widespread knowledge among policy makers about how reliable impact evaluation should be undertaken.

The recently-published OECD Framework for the Evaluation of SME and Entrepreneurship Policies seeks to address these issues by making two major contributions. First, it sets out how to achieve reliable impact evaluation in SME and entrepreneurship policy, including offering a six-step scale to judge the reliability of any evaluation or evaluation method. Second, it provides a summary of what is known about policy impacts from international meta-evaluations and a selection of reliable individual evaluations in the field, spanning a wide range of policy areas and OECD countries.

Randomised controlled trials (RCTs) sit at the top of the OECD’s six-step scale as the most reliable method for impact evaluation. In the context of entrepreneurship policy, RCTs refer to experiments in which entrepreneurs are randomly assigned to different forms of treatment. This can often involve assigning one to a treatment group – which receives the programme support – and others to a control group, which does not receive support. However, in practice many policy experiments will instead make comparisons between different forms of support. By taking account of selection bias, RCTs are widely deemed the Gold Standard for ensuring that any observed improvements in the performance of firms or individuals within a programme are legitimately attributed to the programme alone. While RCTs are becoming more common, important questions remain regarding how best to design and implement RCTs. In addition, RCTs are distinct from other evaluation because they require public support to be allocated on a randomised basis, which has implications on the political viability of this approach.

Why attend the webinar?

This event will bring together policymakers and evaluation experts to share knowledge and lessons on conducting high-quality and reliable impact evaluation of entrepreneurship policies, with a particular focus on RCTs. It offers the opportunity to get involved in debate and discussion with a range of constituents from the OECD, GEN and IGL communities on this important topic. The issues that will be covered include:

• What is the role of impact evaluation in the context of entrepreneurship policy?
• How can RCTs be used to evaluate the impact of entrepreneurship policies?
• What are the advantages and limitations of RCTs? What are the barriers how can these be overcome?
• How can the results of impact evaluations be integrated into policymaking?

Agenda

Event moderator: Matt Smith, Director of Policy & Research, Global Entrepreneurship Network

• 2:00 - 2:10: Introduction to the webinar
  Dr. Jonathan Potter, Head of the Entrepreneurship Policy and Analysis Unit, Centre for Entrepreneurship, SMEs, Regions and Cities, OECD
   
• 2:10 - 2:30: Increasing the impact of innovation and entrepreneurship policies by establishing robust evidence on policy impacts
  James Phipps, Deputy Director for the Innovation Growth Lab, Nesta
  
  Presentation on how to increase the use and impact of robust policy evaluations to facilitate the development and implementation of more effective entrepreneurship policies and programmes. This will cover the IGL’s work in building an evidence base on the impact of different policy interventions, with international examples of IGL-supported RCTs to evaluate entrepreneurship policies.
   
• 2:30 - 2:50: Practical insights from a randomised control trial
  Professor Elena Novelli, Professor of Strategy, Bayes Business School

  Presentation on the experience of a selected OECD country in conducting a randomised control trial (RCT) to evaluate the impact(s) of an entrepreneurship policy or programme. The presentation will cover:

  • The evaluation methodologies deployed for the RCT;

  • The findings of the RCT and the implications for policy;

  • The use of the RCT findings in future policy decision making;

  • The challenges faced and lessons learned from conducting the RCT.

• 2:50 - 3:20: Panel discussion: Insights and perspectives from policymakers and experts
  - Matt Smith, Director of Policy & Research, GEN (Moderator)
  - Nir Ben-Aharon, Head of Policy and Research Department, Small and Medium Business Agency, Ministry of Economy and Industry of Israel
  - Ana Goicoechea, Senior Economist, World Bank Group
  - Albert Bravo-Biosca, Director for the Innovation Growth Lab, Nesta
  
  This discussion will involve national policymakers and experts making points and taking questions from the moderator. Audience members will also have the opportunity to submit questions to the moderator.

  • How can randomised control trials (RCTs) be used more widely and effectively to better evaluate entrepreneurship policies?

  • What are the barriers that policymakers and evaluators face in conducting RCTs?

  • What alternatives exist to RCTs?

  • How can the results of impact evaluations be integrated into policy making?

  • What are the different types of experimentation, and where could people start?

• 3:20 - 3:30: Closing remarks
  - Dr. Jonathan Potter, Head of the Entrepreneurship Policy and Analysis Unit, Centre for Entrepreneurship, SMEs, Regions and Cities, OECD
  - James Phipps, Deputy Director for the Innovation Growth Lab, Nesta
  - Matt Smith, Director for Policy & Research, Global Entrepreneurship Network