A few years back, I worked on a three year contract supporting organisational change in a group of universities who were starting to come to terms with a then brand new agenda, where academics and researchers needed to become more outwards facing, connecting with the public on their doorstep and at large. Part of my role was to mentor internal directors and project managers, and the departments they worked with, in looking for the impact of their activities. Like many major programmes, the initiative had a quite intense, technical, formal, robust evaluation system underpinning it. Like many organisations, this was not the fun part of anyone's work, and on top of everything else going on, was not generally what most people were interested in prioritising. In my mentoring role, I wanted to increase people's confidence about being able to carry out evaluation that was realistic and meaningful, and reduce their fear of becoming overwhelmed. At the same time, some of the community groups involved had been saying their previous experiences of evaluation in university programmes had, at times, been overwhelming, invasive, and one-sided.
​As a result, I created a simple, practical, set of suggestions to make evaluation do-able, useful, positive and meaningful. It simply offered these ten top tips, and five years later with the huge amount of learning I've developed about evaluation, they still absolutely stand the test of time...

1. Be Selective: stay focused – keep your aims and objectives in mind. Don’t try and capture everything about everything. 
2. Impact: always aim to answer these questions: What happened that wouldn’t otherwise have happened? What difference did it make – to us and to our participants, partners or colleagues? What did we learn from this? What do we intend to do next?
3. Achievability: think about what’s realistic and achievable with the resources you have, this may mean you select only one thing to explore in depth, and a few other areas to cover briefly. 
4. Breadth and depth: include quantitative and qualitative information. Quantitative methods lead to facts and figures; qualitative methods capture experiences and personal impacts. 
5. Varied questioning: use open and closed questions. Closed questions allow you to create statistics to show particular patterns of what’s happening. Open questions mean people can say what’s really important to them. 
6. Bespoke and appropriate: every project or event is unique. Create methods that work for you and your partners/participants.  Think about what will make it easy for people to complete any evaluation requests and, where appropriate, make it fun and integrate it into the main activity of your work. 
7. Combined approaches: where possible combine a range of evaluation methods and techniques. This will allow you to cross-reference your findings to develop a clearer understanding of what is happening. 
8. People-centred: Remember to explain to participants why you are collecting the information and be mindful of protecting private information. 
9. Analysis: analyse the information you have.  Statistics and conclusions are just the start, keep going until you know why is that important? What does it show or prove?  
10. Applying the understanding: use your evaluation. Tell people what you found out and the lessons you have learned. Consider how it will inform future plans and strategies going forward.  Reflect on how you might do things differently next time.

I've just recently started work on the evaluation of a year long programme hosted by Manchester Metropolitan University's Institute of Humanities and Social Science research. Entitled Creating Our Future Histories, the scheme sees 'early career researchers' (usually those who are completing a PhD, or are just about to start one / have recently finished one) working with Manchester community organisations. Each partnership is mentored by a more experienced academic. The partnerships are punctuated along the way by a series of weekend workshops combining into a professional development course on how community engagement between academics / researchers / communities might take shape. Each partnership is also expected to meet at least once between each workshop.

The partner-groups are developing co-constructed plans and activities which research previously uncharted areas of the organisation's heritage, and look towards incorporating their future in a way which will become part of their heritage in years to come - there's the 'Future Histories' part. Late next Spring each group will showcase their findings in creative and public ways - many yet to be decided; though ideas are already circulating about film, video, exhibtions, time capsules and more.

I'm about a month in and I'm once again struck by the many rich and hidden histories of Manchester - industry, architecture, battle and radical action, many many things which show the inventiveness and resilience of this sometimes bloody minded and often ingenius city.

You can find out more about the project here and I particularly recommend the research group pages and project blogs to find out more about the organisations involved and the progress and reflections taking place.

I'm just wrapping up some research for a museum. They asked me to collate case studies of good and innovative practice in how comparable venues (which in this case include medium-large scale museums and galleries) use digital technology to support school visits, in workshops, self-directed studies and potentially back in school.

They also wanted to find out about the ways such activity can be evaluated. They absolutely do not want to have form after form handed to teachers and students, and wondered how else really good evaluation might take place.

The brief contains phrases like blended learning and e-learning. It's problematic because there are no clear definitions of what those are and where they start and end, And it's a real rabbit hole - an entire and massive area of specialisation and expertise.

It's a small piece of work, just skimming the surface to help the museum think in new and different ways about what they might do, and how to monitor its impact well.

I've collated 64 pages, over 32,000 words, of case studies of applications, programmes, projects, reviews and industry expertise opinions on contextual issues such as evaluation, future proofing and general good practice in digital learning and engagement. I've visited more websites, read more conference papers, searched more forums than I can count and interviewed some really insightful and inspiring colleagues in the field.

Eventually, if the museum in question has no objections, I'll upload the collated set of case studies and expertise here for anyone else who might like it. It will be in a very rough and ready format - just my research notes really, in no particular order. But it may be of some help to someone so watch this space...

In the meantime, it seemed much easier to put all 32k+ words into wordle and see what happened. There it is above, that's what the whole shebang amounts to. Interesting at this stage that 'online' is so prominent, given that I wasn't specifically looking at just online options. Interesting too that if 'conversation', 'collaboration' or 'participation' are in there, they certainly don't jump out.

I'm now part of the network exploring the SROI model of social returns on investment. It's way of putting monetary values on the sorts of evaluation and participation outcomes that occur in the projects I work with, and demonstrating what difference the investment has really made - what happened that coulnd't or wouldn't have happened without.

It's also a way for me to offer organisations the sort of robustness they might expect from an academic research team. 

A lot of it will be new to me but I'm hoping it will ad a layer of technical formality to the likes of MLA's Generic Social Outcomes. Indeed MLA have already commissioned large scale pilot evaluation projects testing SROI's capacity for analysing the values of specific library and museums schemes.

Exploring qualitative outcomes, impacts and benefits is something I really enjoy. There are times though, when stakeholders need more quantifiable results and this should enable me to provide that too.

SROI has been developed, and continues to be developed by it's members, to support arts and culture, public services, science and education - all of which are fields I work with. It also supports employment and business, environment and climate, and health and care. So I hope this means it will become a tool that is recognised across all organisations.

I should add though, that for me this will be an additional way of identifying what works and where things can be improved. It won't be the only one. I don't think any single tool can really capture everything that's important about a project. It's important to create a variety of tools bespoke to each separate project to cross-reference information and help pick out trends, similarities, patterns but also individual stories - the illustrations of people who have really felt something change as a result of taking part. But this does offer a way to  create evidence of quantifiable outcomes alongside the qualitative benefits I'm always keen to advocate based on evidence.  

Interestingly DEMOS have just released a report suggesting that the approaches underpinning SROI are sound though a range of shared outcome measurements across the public sector, which can be gathered more simply than SROI is able to, is desperately needed in order to help smaller organisations demonstrate their worth. I can't argue with that kind of logic, but until it arrives, SROI seems to be the closest there is to different sectors speaking to the same language.

It's the start of a new learning journey for me, and a welcome addition to what I will be able to offer the organisations I work with.

In evaluation, looking at outcomes is vital. When you've invested in a project of course you'll need to know what was achieved and where there are still gaps. But it can be cumbersome reading mounds and mounds of text in a report. Charts, graphs and percentages can help clarify, but can still make for dull reading (and for some feel too much like maths homework).

To help busy partners to any project, finding a variety of ways to report on activity can make all the difference between people remembering what went well and being able to advocate the value of their work... or not.

I've recently been creating a toolkit for early years practitioners looking at how creative engagement can help achieve new and unexpected results. Its aim is to be quickly digestible and highly practical. The toolkit is based on the activity of ten creative early years projects in schools and Children's  Centres in the North of England.

The projects were mapped against the Early Years Foundation Stage six areas of learning. To illustrate which of the EYFS outcomes the projects really brought to life, I used wordle to create this at a glance illustration. The larger the word, the more presence it had in the projects. You can see here Personal, Social and Emotional development was the strongest feature across the programme overall.

It doesn't replace the need for writing other information in the report of course. However the teachers, children's centre staff and creative practitioners involved, and readers of the toolkit, can now instantly see where the projects thrived and what kinds of outcomes similar work might expect to achieve, so much more quickly and easily than deciphering a big chunk of writing or trying to analyse a graph.

There are some things I've thought it would be good to do for a long time, but it's never been part of my paid work to do them, and so they haven't got done. I'm noting them here as a reminder to myself and an invitation to anyone who is interested, to commission me to do  them, and share them freely and publicly (or at least under creative commons) for the wider benefit of those who can use them:

1. A list of what managing, researching and evaluating many projects has taught me about partnership - what it does or doesn't mean, what it looks like, the different shapes it takes, the benefits, challenges, pitfalls, successes, opportunities and more

2. A top ten of findings / recommendations that arise in my evaluation work. Mostly around the process of managing projects - resources, processes, staffing, communication, planning, sustainability and legacy - those sorts of things. There are so many shared benefits and challenges I see time and time again, to the point it makes me sad that these things have to be learned afresh each time with publicly funded activity. Wouldn't it be better if we can do more learning from each other and move things on?

3. A portfolio of what creative consultation / creative evaluation can look like - examples of things I've tried myself and examples that other people have developed in their work.

There might be more to this list of lists but these are the freshest in my mind just now. So if you can't afford a specialist consultant for these things, do keep popping back, maybe one day someone will have commissioned them from me and they will be available to you for free.

Unintentionally I have carved out something of a niche in my repertoire, in the form of evaluation. I'm not sure if its because I naturally love to disect and deconstruct what is seemingly apparent to see what else is really going on; or whether it's simply because other people don't enjoy it and are happy to hand it to someone else. (I'm lucky in that much of my evaluation work comes through recommendation).

I know at times it's because organisations know in great detail so many things about what they do and the way they work, but need to be able to show that they have explored this objectively, and so hire an external consultant.

Often I never know how much it influences organisations to change, adapt and experiment, though I have one client who I know does read them with a fine tooth comb, come back to me query and delve deeper into some parts, and act on it annually, which is very rewarding indeed.

For reasons not exciting enough to share here, I spent much of today re-reading past evaluation reports and realising how fascinating this process of reflection really is. It was refreshing to read these again, with space and time, for what they really say, rather than in a proof reading and contract fulfilling, deadline looming frame of mind.

An introduction to evaluation with some examples of how to avoid too much form-filling and tick-boxing, thank goodness.