Acting Fast or Slow?
What we can learn from product design principles.
Hello and welcome to Good Evidence. If you are new and you would like to stay up to date with the newest posts, why not subscribe here:
As you can tell from the name, I started this newsletter to investigate the question: why isn’t evidence being used more to make (particularly economic) policies?1 And it isn’t that I think what we are doing isn’t good, it’s just that I think the benefits of doing even better could make such a difference.
But am I right to see that as a problem? Maybe it’s just me who thinks that - as someone who spent a lot of time thinking and working on evidence, it is natural for me to want more being used. But I have enough self awareness that I realise this doesn’t mean it’s right.
So what is the best way to go about this? At a basic level, designing a policy is just like designing the solution to any problem. And there is actually quite a lot of thinking on the this in other disciplines, such as product design. Looking at these can be really helpful when thinking about how to design policy.2
So today, I want to take a step back and think about how the way policy is usually designed compares to best practices in product design. And particularly, are we spending the right amount of time on the right things?
How to design anything - lessons from product design
Here is a basic diagram of the broad steps in designing a product:
Mostly, the steps are very common sense: You start by engaging with potential customers to understand what they want (empathize) and clearly define the problem or need (define). Then you brainstorm potential solutions (ideate). After that, you choose one of your solutions to develop it further into a prototype (prototype) which you then test with your audience (test). Finally, given the feedback from the audience, you do one or two things: you either make small tweaks to your prototype and test again OR you realize you need to adjust your understanding of the problem. In that case, you go back to the drawing board and walk through the process quickly again. Are there any new solutions to brainstorm? Do you now select a different solution or do you just adjust your prototype?
Ok, so this sounds quite obvious. If you asked anyone to think about how the process should work, they would probably come up with this. But the key emphasis of this framework is not the steps but how you complete them.
Firstly, the framework asks you to do each step after another. This has several benefits. Spending time understanding the problem without bringing in solutions allows you to enter the discussions with fewer preconceptions. Focusing only on coming up with ideas before thinking of how you would implement them encourages out-of-the-box thinking and keeps you from overlooking new ideas.
Secondly, it recommends focusing on external feedback over internal thinking. Instead of spending a lot of time picking the perfect option or designing the perfect product, you prioritize the external facing parts of the process: understanding the problem and testing the product. Everyone is subject to optimism bias and so will overestimate their understanding of what will work. It will be very tempting to spend lots of time on the part where you do all of the thinking hoping to get it right that way. Instead spend the minimal time on this.
Finally, the whole process is designed to be done over and over. You shouldn’t complete a step and then never look back. Instead, the idea is to move through the process relatively quickly and then revisit things after you get the feedback. Ideally, you designed your testing so it doesn’t just tell you about the impact of the solution but also about the underlying problem. Which means, designing your testing is as important as designing the solution. #monitoringandevaluationmatters
The reason this requires definition in a framework is not that it is novel, but because it is very hard to follow. It is so tempting to rush into action and start working on solutions. Or to tinker for a long time on your solution until you think you have got it right before sharing it externally. Or to dive into designing before you really understand the problem, etc. etc. Formalizing these steps helps to fight our behavioral biases and reduce the risk of getting it completely wrong and only finding out at the end.
Is this helpful for policymaking?
The above is pretty common sense and I am not the first to catch onto it. The overall steps are the same as described in any guidance on how to design policies.
If you apply these to policymaking, it could look something like this:
But, the interesting thing is not that you do the steps but how you do them. And there are a few ways in which policies are different from products that mean this is a little bit harder to do.
#1: Asking people is not enough
The nice thing about designing a product and not a policy is that if you want to understand what problems your end users face, you can usually just ask them. Maybe they don’t fully realise it straight away but after some probing and suggestions, they often will be able to tell you what they are looking for.
Policy making is different, because policies usually create winners and losers. So, no single person will be able to tell you what the right solution is, because while the policy will hopefully solve some problems it will probably create others. Someone will always be for and someone will always be against your solution. It also means that there is a real risk of getting bogged down in endless discussions that never come to a resolution.
So you need other ways to assess what the problem is and how well our proposed solution fixes it. Most of the time that does include asking people but also looking at other data and evidence- which is why people like me have a job!
Unfortunately, the fact that you may get contradictory information makes some people skeptical of spending any time exploring external information to understand the problem. But that just makes it harder - not not worth doing.
#2: There is a lot of pressure to act quickly
Quite often, in my experience at least, there is pressure to act quickly as soon as a problem has been identified.
Going slow is unpopular with both politicians and voters. Politicians and senior policymakers are often under a lot of pressure to make an announcement of what they are going to do to solve an issue. The person that diligently solves the problem just doesn’t have the same appeal as the person that is willing to be decisive and spring into action.
And there are good reasons for acting quickly. You want to build momentum and not get stuck in endless circles of discussing the issue that never reaches a conclusion. You don’t want to open a door for special interest groups to slow you down. You don’t want the people affected to wait any longer than necessary to get help.
But it can be equally problematic to dive in too quickly. Let’s take an education reform as an example. Implementing a reform that is ineffective is very costly. There may be direct monetary costs of implementing the reforms as the schools need extra funding to implement them. Students’ learning could be disrupted as teachers need to learn the new material/ way of teaching. People will make long-term life choices that cannot be changed easily when the policy is abandoned. So you want to be sure your policy is going to be effective before making the whole country or state change the way you teach.
#3: Institutions are built to scrutinize the option chosen
I worry that what I’ve written so far makes you think that there are no checks in policymaking. That is not true. There are lots and lots of them and they take up a significant amount of time. Parliaments for example don’t just pass laws on a whim, there are committees that try to understand the actual proposals. And they ask a LOT of questions. Not that I’m complaining, it is a really important part of the process.
There are also clear points in the process when policymakers need to ask the public for their opinion, both at the start and at the end. And most countries have made clear monitoring and evaluation commitments for their policies.
But these points of scrutiny tend to focus mostly on discussing the option the policymakers put forward and take the evidence provided at face value. For example, there is little scrutiny if enough was done to understand the problem or whether all options have been explored. And when it comes to engaging with outsiders the mechanisms focus again on asking people for their opinion, which as discussed is just not enough to fully understand.
This can create a system where you find evidence to evaluate your solution after you have basically decided that it’s the best one. That would be fine if people were fully rational. But there are so many behavioral factors at play that means that this is just not as effective as starting with a blank sheet of paper.
#4: It’s hard to go back to the start
Trialing does not convey decisiveness in the politician. Imagine having to go back to your voters and say: remember this policy that we thought was the best thing ever turned out to be not as effective as promised and will now be replaced by a different policy. That’s not going to go down well.
In fact even setting it up as a deliberate trial creates a bit of a trap: either you admit that you don’t think this is the perfect solution yet, in which case how can you defend imposing it on people? Or you say you do think this is the best solution, in which case why aren’t you sharing it with everyone?
All of this means that more often than not policies get implemented fully instead of tested in a (dare I say randomized) trial. There are also policies like laws that cannot be piloted for a smaller group - it sort of goes against the principle to change the rules for some and not others.
This won’t work for me…
Objection #1: Sometimes you just need to move fast
Of course, I get that. Sometimes circumstances - like COVID for example - require you to move incredibly quickly and you can’t spend your time dilly-dallying like a toddler putting on his shoes while there is a global meltdown.
But those kind of policies are often designed to be short term anyway, and it is much easier to build in review points to adjust the policy. In that case it is important to make sure you have a way to check if your policy is working. Which may seem a bit crazy given that there is an international emergency going on. But if you don’t do it, then you find yourself in the same position a few months later still not knowing anything more. And it may not mean running a trial
Objection #2: Some things you just can’t trial
On the other side of the spectrum are policies that are very hard to trial. I go back to my example of the interest rate. You cannot really increase the interest rates for some parts of the economy and not others. In that case, I guess all you can do is really spend time gathering as much evidence as you can so you can really understand the problem. So that way you are more likely to get it right in the first place.
So what should we do?
I don’t have any solutions to the issues above. I think for me the key change is to really prioritize that external engagement. Not to see it as some add-on or someone marking your homework, that ideally is avoided but really as part of the process that clearly feeds into new decisions. And at the same time make sure that we have the best tools to understand and to test.
Now, I am mostly talking about economic or social policies. This is not the same in other areas and in all institutions. Medical guidelines for example are (nearly) all the time based on empirical evidence and developed closely with experts who evaluate the existing evidence.
I’m not the only one who thinks that - as demonstrated by training I attended a few years ago that inspired most of this article called product design principles for policy making or something to that effect.