Where is the real value in charity impact evaluation studies?- Written by Amy Franklin
For nine years, I worked in corporate fundraising, in that role I regularly used evaluation reports but my focus was very much limited to extracting latest headline figures and good news stories that would sound appealing and could be used to generate more funds. I don’t apologise for this - it helped me to make more money for the organisations I worked for to put back into our services – but it wasn’t until five years ago, when I started in a service delivery role at Walking With The Wounded (WWTW), that I really appreciated that evaluation doesn’t end with those soundbites from a report.
Instead, an evaluation report should be just the beginning of a long and often complicated journey that is ultimately about building our understanding of what works in tackling some of the most complex and challenging societal issues.
WWTW run a number of programmes designed to support veterans in the community. These focus on those with physical and mental injury, and individuals who have struggled in their transition out of the military. Some fail to adjust to the civilian world, and this can lead to social isolation and often-complex needs. Our programmes help veterans return to the workplace and provide long-term security for themselves and their families.
As most charities would recognise, there are few quick wins for solving the challenges our service users face. However, robust evaluations of our programmes allow us to continuously improve our effectiveness by shedding light on what’s working and, importantly, what’s not working so well, so that we can test new approaches and compare them to see whether the result is as we hoped.
Back in 2016, WWTW funded an evaluation of our services, which was independently carried out and analysed by the King’s Centre for Military Health Research, King’s College London.
This evaluation aimed to provide information to enable us to improve upon the support WWTW provide ex-service personnel. The researcher looked at three of our support programmes: Head Start (mental health), Employment and First Steps (training).
An impact and process evaluation was chosen for their investigation, a model that aims to determine whether programme activities were implemented as intended and whether they achieved the desired results. The researcher gained quantitative information with the use of standard assessments. These were completed with service users before and several months after the service was delivered by WWTW. This was supplemented with a series of qualitative interviews to explore the processes of programme engagement, in other words; what happened, what went well or did not go well. These interviews were conducted with both service users that completed treatment or a course of support with WWTW, and those that disengaged.
Over the course of three years, a total sample of 283 beneficiaries engaged in the three WWTW programmes, with 172 completing, 61 dropping out or disengaging and 31 still ongoing at the time that data collection ended in February 2018.
The research found that Head Start was a clinically effective programme for anxiety symptoms, leading to improved mental health and functionality. This is particularly valuable for employment achievement and maintenance. Qualitative data revealed that overall, Head Start was highly regarded and perceived to be beneficial. Qualitative feedback on First Steps and the Employment Programme was also positive. This feedback showed that our programmes have a positive impact on the veteran community.
Several recommendations were made about how we could improve our offering.
As of August 2019, we have updated the referral paperwork and the extended referral form is now available on our website. We also now routinely follow up with all clients who complete therapy at 6 and 12 months, and repeat the same clinical measures we used pre-therapy, so we understand how that client’s mental health has improved, or not, throughout therapy and beyond. Over the next year this will enable us to measure whether the other recommendations addressed have had an impact by comparing them to the follow up scores we achieved in the PHD research. It was also recommended that we conduct a treatment review after 4 funded therapy sessions to establish the appropriateness of the treatment type and assess the need for later top-up sessions. This is something that we are starting to roll-out. Lastly, we were encouraged to think about holistic care for those we support, in line with the idea that addressing a problem in isolation is a limited approach when a service user has multiple needs. Whilst we cannot achieve this level of support in all our programmes across the UK, we are piloting a programme in the . This will provide support to veterans in serious stress, their families and carers by addressing all their support needs in partnership with a group of regional delivery partners. We aim to support 225 individuals over the next two years and thoroughly evaluate this approach.
Whilst we were pleased with the conclusion of the academic evaluation, this was really just the start. We will continue to make adjustments, evaluate, review and refine as we look to improve our programme.
I look forward to updating you again in the future. In the meantime, I would like to leave you with David’s story.