Three approaches to evaluating social innovation – which is right for you?

However much you know about the environment you're working in, there is a way to evaluate the effects of your intervention
social impact charity
What is the best way to measure social impact? Photograph: Robert Llewelyn

People love to argue about the best way to assess social impact. For example, whether you should have an external evaluator, or do a randomised control trial (RCT). So what is the best way to do an evaluation, especially if you're working in social innovation?

The important thing is to choose the right evaluation approach for the stage of development of your intervention. One way to do this is to use a scale showing the stage of development of your project.

On the left hand side of the scale are early stage or pilot projects. At this point you are still developing, or possibly even beginning to create, your model and approach. You are probably still not sure of the effects of your work. You need to consider reflective evaluation approaches that enable you to gather data and understand the types of effects you are having, both positive and negative.

Cast the net wide – ask open and diverse questions about the different areas and people your work could affect. If you just test for specific things, and go for rigid outcomes measures, you will miss out on capturing unforeseen information. At this stage you want to be focused on communicating the effects of the work internally so that you can better improve your practice and delivery.

At the extreme right of the scale are established interventions with a replicable model. This sort of evaluation is frequently outward facing as it's concerned with capturing and articulating the social value of your work to others.

At this stage, because you have already developed a good idea of the intended and unintended consequences of the work, you can be more selective about the specific outcomes you want to measure to best articulate the value of your work. You will want to use the most rigorous methods to do this, like RCTs to deal with issues like attribution.

If you are developing a social innovation, which by its nature deals with an unknown space, you can't possibly start here. Though, in looking to scale and replicate, you should certainly be aiming to get here over time.

A good way to assess what evaluation approaches are appropriate at different stages and why is to use the Cynefin framework from Cognitive Edge.

This divides the evaluation into three types – simple, complicated and complex spaces. These all relate to how much you know about the environment in which you're working.

Simple space: When we have a significant amount of research about the context and effects of an intervention, we are in the simple space. That is, we are pretty sure that doing 'x', will have 'y' effect. Nothing's ever guaranteed of course, but it's a good bet. Here you can focus most of your measurement on what you think will happen. You can consider using approaches like RCTs, because you know the effects you're looking for, and you need the most robust method to detect them.

Complicated space: However, sometimes, although you know the context you're working in quite well, there's still debate and decision about the best way forward. You might have several different possible paths to take to address a social challenge that all seem feasible. This is a complicated space. Here you should focus on measuring what you think will happen, so measure your sought outcomes.

However, you need to consider the efficacy of different approaches. You might consider a/b testing, where you run two slightly different versions of an intervention and see what gives the best results, or comparison group studies. Also you still need to be mindful of the unintended consequences of your work so you should have systems in place to capture those things that don't show up with established measures.

Complex space: Here we have places where we're developing new interventions and don't know how or why they affect change. This is the complex space and is where social innovation happens. (After all, if we knew what was going to happen, it wouldn't be innovative.)

You need evaluation approaches that can detect the unforeseen and unexpected effects of your work. Here you need to think about open captures that are not restricted to looking for change according to specific outcomes. There are myriad techniques, but at the moment the Cognitive Edge approach is the most effective in my mind.

What's crucial is to recognise that no one evaluation approach is better than another. It's about picking the right evaluation approach for the stage of development of your intervention, and that is the approach that offers you the information you most need at that time. Early stage ideas need more open data captures because you are still informing the development of your intervention and trying to understand its effects. Later stage projects should have this information, so are more interested in testing for specific impact of the work.

Kieron Kirkland is development research manager at Nominet Trust.
