Evidence of impact is an important part of what funders look for in applications – but there’s more to it than that, says Kieron Kirkland
We’re increasingly being told that evidence of impact is crucial to successfully getting funding. However, the Alliance for Useful Evidence’s latest report, The Secrets of Success, shows us that there’s quite a gap between the rhetoric of charities needing to provide rigorous evidence of impact, and the reality of how funders make their decisions.
The fact is, funders will never base their decisions on evidence of impact alone. While evidence is crucial to moving forward and supporting effective interventions, it is not, in itself, enough to make decisions. Life is chaotic, messy and unpredictable. There are countless examples of projects that have been funded, often for huge sums of money, but that have failed because of key unseen factors.
Unknown quantities
One of the great nonsenses of the funding world is that organisations are expected to know exactly what’s going to happen at the beginning of their project. An idea will shift and change as it adapts to a new and changing context, unforeseen problems arise and new opportunities present themselves. So while an idea may sound good and have amazing evidence, funders usually know that’s not all there is to it. They will be looking for two key things to minimise risk: the team and the host organisation.
The Secrets of Success report found that ‘tacit knowledge’ about delivery teams is a big part of funders’ decisions. There’s a good reason for this: it’s the team that is going to be dealing with and responding to all the problems and changes as an idea develops. While an idea may change, the team will generally stay constant and deliver it (particularly if you’re dealing with early-stage social ventures), so funders will usually consider the skills and experience of the team it is investing in, as much as the idea itself.
Similarly, if there is a strong organisation behind the project, this also has the potential to take a lot of the risk out of a grant or social investment. From experience, this tends to be more important for traditional grantmakers. If you’re a new organisation, it’s a good idea to build up your profile, perhaps through smaller projects, or by highlighting previous expertise and experience to help build confidence in your work.
Applications for funding should articulate the specific skills and experience that both the team and the organisation as a whole bring to the work. Remember this is about proving skills and ‘de-risking’ the work, so think about how different team members support specific aspects of the projects development and delivery.
It’s always worth keeping an eye out for opportunities to meet funders, too. If you’re trying to get people to believe in you and your project, it’s much easier if you’ve already made a personal connection. The benefits of networking cannot be underestimated.
All shapes and sizes
Another key reason why funders look beyond evidence of impact when making investment decisions is that many of them deal with very diverse applications, from projects at varying stages of development, in different sectors and from organisations of different sizes. This means there’s very little they can consistently look for when making comparisons about the relative strength of the evidence offered by an organisation or their impact strategy.
This problem is compounded by the fact that there is no single ‘best’ way to measure your impact. Different measures are appropriate at different stages of development of a product or service. Furthermore, it’s unlikely that your funder has an extensive understanding of impact measurement; at best, there are only a few impact measurement specialists scattered among the very big foundations, so don’t assume the person reading the application is an expert.
One thing we regularly hear from social investors and foundations is that they would like prospective fundees to measure their impact consistently, with a robust system in place, whatever the approach happens to be. The bedrock of this is good data. If you haven’t got decent data collection systems in place, everything else falls apart.
In reality, no method of impact measurement is a panacea. Don’t leap on the latest evaluation buzz word, or assume that’s what funders want. The latest trend is for Randomised Control Trials (RCTs); it was Social Return on Investment (SROI) not so long ago. But this is potentially crippling for charities that are expected to invest resources in these new systems. It also damages the integrity of the measurement approach itself, by forcing it to be used at inappropriate times. There will be occasions when an RCT is appropriate; at other times, a cost/benefit analysis will be right. Sometimes, you’ll simply need to get some systematic data collection about the people you’re working with before going further.
Whatever method you use, build in the time and budget for your evaluation and impact measurement from the start. The Charities Evaluation Service recommends you put aside 5-10 per cent of your budget for evaluation, and funders themselves will sometimes specify a percentage that must be allocated.
When allocating your budget, bear in mind that different evaluation methods are appropriate for different stages of an intervention. Early-stage projects need more process evaluation to understand what’s happening. Later-stage projects can engage with specific approaches like cost/benefit analysis.
Eye of the beholder
Each funder will also have its own set of values and its own theory of change. At Nominet Trust, we will assess an application not just on its social value (the change it will make), but also its user value (will people actually want to use the product or service) and its market value (will anyone actually pay for it and keep it sustainable).
So, think about what your next funder or investor will want to see, and how they measure value. An education audience, for example, is going to care about attainment, whereas governments are generally keen on cost/benefit analysis.
The bottom line is: know who your audience is and what, exactly, you want to measure. Where possible, use independently verified approaches. This could mean using an external evaluator, but it might also mean simply using established measures. If you are measuring wellbeing, for example, don’t just make up your own questionnaire – use one of the many established scales, like the Warwick and Edinburgh Wellbeing scale. The Charities Evaluation Services (www.ces-vol.org.uk) and NPC (www.thinknpc.org) websites contain many similarly useful resources, so make use of them.
Opening up
One of the great contradictions underpinning the funding world is ‘sharing failure’. Charities often don’t want to admit when something has gone badly because, very logically, they’re worried about the impact on future funding. But if no one shares their evidence of failure and the lessons learned, we end up in the same circle of funding things because they sound like a good idea, rather than because they are actually effective.
To overcome this issue, at Nominet Trust we separate monitoring from evaluation. Monitoring relates to the organisation’s project delivery; evaluation is about understanding the success of the model. This approach recognises that an organisation might have done everything it said it would, but the evidence shows that the model just doesn’t work. By separating monitoring and evaluation, we ensure that we’re testing models, rather than organisations.
As a sector, we need to move away from a closed approach to learning, and start openly sharing lessons. If we are to do this, charities need to feel that by being honest, they won’t be penalised by foundations. Foundations, for their part, need to open up and embrace learning. At the moment, they are sitting on a wealth of information and, mostly, it stays there.
Foundations can begin by asking their projects for monitoring and evaluation information that is inherently more open. One way of doing this might be to ask projects to document their progress through regular blogs, or by video, instead of using rigid monitoring forms. Of course this needs to be appropriate; sometimes charities will want and need to keep things private. But sharing should be the default, rather than the exception, and foundations can lead the way through modifying their reporting processes.
Encouraging and enabling sharing will have several benefits for charities. Firstly, it will dramatically widen the information available in the sector. Secondly, it will help signpost that a particular charity is a thought leader in a field (that they’ve been there and done it, so to speak). Thirdly, it means that charities’ work and reflections will be openly available for potential future funders.
Too much information?
While a system of open data is undoubtedly the way forward, this data will have to be structured properly if we don’t want to run into problems. Masses of blogs, videos and other media could lead to us being awash with information. For it to be really useful, we need systems to aggregate the information and find patterns in the data. At Nominet Trust, we’ve been trialling a new approach that collects large amounts of beneficiaries’ stories and allows them to be aggregated and analysed on a big scale, using an innovative tagging approach. We’re also developing data mining tools to make sense of monitoring and evaluation reports.
On a practical note, if you’re using blogs and videos, try to keep them on one platform and use a tagging system so you can search and keep track of your archive.
So, is evidence important? Yes, but there’s more to funders’ decisions than that. Funders and investors and look at more than just the idea when deciding on grants: they look for a good team or organisation, robust impact reporting and sound data management, among other things.
Ultimately, charities and funders need to be partners in enabling social change. Evidence should support that partnership, not derail it.
Kieron Kirkland is development research manager at Nominet Trust
This article first appeared in The Fundraiser magazine, Issue 34, October 2013
