Test + Transform: Policy Experimentation to Elevate Your Impact

Session Description 

Looking to take your programs to the next level? Policy experimentation is a game-changing tool for ecosystem support organizations and government agencies aiming to drive innovation and improve effectiveness. By testing new ideas in real-world settings, you can identify what works, minimize risks, and achieve better outcomes. In this interactive workshop, we’ll leverage the expertise of the Innovation Growth Lab to share real-world examples from around the globe - and help you explore fresh, actionable ideas and solutions to your own challenges. 

Join us to:

• Unlock the Power of Policy Experimentation: Learn how testing new ideas in real-world settings can improve your programs, minimize risks, and lead to better outcomes for entrepreneurs and startups.
• Transform Your Programs for Greater Impact: Discover actionable strategies to enhance your initiatives, whether you're focused on training, funding, or bridging industry and academia, using data-driven approaches and experimentation.
• Global Insights and Real-World Examples: Benefit from the expertise of the Innovation Growth Lab and case studies from around the world to understand how policy experiments can drive innovation and improve effectiveness.
• Collaborative Problem-Solving: Come prepared to share your challenges and brainstorm solutions with peers in an interactive setting, gaining fresh perspectives and actionable ideas to apply to your own work.
• Make a Tangible Impact: Walk away with the tools and mindset needed to rethink and redesign your programs, leading to measurable improvements in your ecosystem's support for entrepreneurs.

Session Recap

Becoming Experimental: How to Test and Learn for More Effective Policies and Programs

Brief Summary
This session focused on the importance of adopting experimental approaches in designing and implementing policies, programs, and organizational strategies to support innovation, entrepreneurship, and productivity. The speaker, from IGL (a global policy lab), highlighted how traditional decision-making often relies too heavily on intuition and stories, leading to programs with questionable impact. The core message was that by integrating data, evidence, and systematic testing, organizations can de-risk new initiatives, continuously improve their interventions, make better decisions, and ultimately achieve greater impact. Participants were encouraged to think about applying experimental methods to the specific challenges they face in their work.

Outline

• Introductions: Participants shared their backgrounds and the focus areas of their work in ecosystem building, economic development, and support for entrepreneurs.

• Session Overview: The plan included an introduction to IGL, the concept of "test and learn" and experimental approaches, the importance of evidence, and a practical exercise on designing experiments for real-world problems.

• IGL's Mission and Approach: Discussed IGL as a bridge between policy and research communities, promoting agile, experimental, data-driven, and evidence-based approaches.

• Critique of Traditional Approaches: Contrasting the "hope strategy" (design and hope it works) with the experimental approach (start small, test, learn, scale).

• Benefits of Experimentation: Exploring how testing leads to new solutions, de-risking initiatives, continuous improvement, time-limited programs, better decision-making, and saving money.

• Becoming Experimental: Key requirements include developing an experimental mindset (asking "what if"), fostering a culture that allows for trying new things and managing risk, and using appropriate methods for learning.

• Experimentation Process: Outlined steps from defining the problem, designing interventions, piloting/testing, to learning and scaling.

• When to Experiment: Applicable in various phases, from understanding problems and testing mechanisms to optimizing delivery and demonstrating impact.

• The Challenge of Evaluation: Discussed the lack of robust evaluations and the importance of using credible methods that go beyond satisfaction surveys to demonstrate actual impact, citing a study on local economic growth schemes.

• The Need for Comparison Groups: Emphasized that robust evaluation requires comparing outcomes for those receiving an intervention against a group that didn't, to avoid confounding effects.

• Randomized Controlled Trials (RCTs): Introduced RCTs as a powerful method for creating comparable groups and reliably measuring impact, drawing parallels to testing in health and social policy.

• Examples of Experiments: Shared case studies including peer learning for businesses in China, facilitating collaboration among researchers, reducing bias in funding applications for women entrepreneurs, and increasing program applications through targeted messaging.

• Operationalizing Experimentation: Addressing practical challenges for small teams or those with limited resources, suggesting collaboration with academics or structuring large-scale initiatives with built-in variations for learning.

• Areas Where Experimentation Helps: Discussed how it can be applied to specific components or a "user journey" of a program, even if not suitable for high-level strategic decisions like sector prioritization.

• Focusing on the Problem and Theory of Change: Stressed the importance of deeply understanding the problem and the underlying assumptions of how an intervention is expected to work before designing a solution.

• Designing an Experiment (Thought Exercise): Introduced the PICO approach (Population, Intervention, Control, Outcome) for clearly defining the research question and components of an experiment.

• Practical Application: Participants began discussing their own problems and potential solutions to frame them for experimental testing.

Notable Quotes

• "Our belief is that policies, programs, organizations are much more effective if they are agile, if they are experimental if they are data-driven and if they're evidence-based."
  "Experimentation is a very useful way to the risk."

• "Get in love with the problem, not with the solution."

Key Takeaways

• To maximize impact, programs and policies supporting innovation and entrepreneurship should move beyond traditional approaches and embrace experimentation.

• Making good decisions requires combining intuition, data, stories, case studies, and evidence, rather than relying on just one or two.

• Robust evaluation is critical to understand if an intervention truly works, necessitating the use of comparison groups and methods like RCTs to isolate impact.

• An experimental mindset and a culture comfortable with testing and potential failure are as important as the specific methods used.

• Before jumping to solutions, thoroughly understand the problem and map out the assumed steps (theory of change) by which an intervention will lead to the desired outcome.

Resources Mentioned

• IGL (Global Policy Lab): The speaker's organization, focused on promoting experimental approaches in policy.

• London School of Economics: Referenced for a study evaluating local economic growth schemes.

• Top Economic Journals: Mentioned as the publication venue for the research on the China business peer learning experiment.

• PICO Approach: A framework suggested for defining the components of an experiment (Population, Intervention, Control, Outcome).

• A handbook on using AI in the peer-review process was mentioned in the context of experiments run by science innovation funders.

Action Items
While no formal post-session action items were assigned, the session included practical steps for participants to engage with the concepts:

• Network and Share: Participants were encouraged to move and sit with others working in similar areas for more productive conversation.

• Identify Challenges/Opportunities: Attendees were asked to think about a specific problem they face or an opportunity they want to explore within a program they run.

• Brainstorm Solutions/Ideas: Participants were prompted to consider different ideas or changes that could address the identified issue.

• Design an Experiment: As a thought exercise, participants began working on converting one of their potential solutions into a basic experimental design using the PICO approach.