Ok here’s how a normal evaluation goes.
Funder decides a project evaluation is due.
Funder talks to project (Optonal- sometimes they don’t ).
Funder and Project decide on questions, time frame and suggested methods.
From the funder’s office the ToR (Terms of Reference) emerges. It clearly states how pre-defined questions should be answered by predefined methods to produce (predefined) answers to be used in predefined ways.
Terms of Reference are circulated. Evaluators apply.
Someone is selected who is deemed to best fulfil that predefined role in terms of qualifications, professionalism, objectivity, experience, cost effectiveness, efficiency.
The evaluation plays out as expected.
A report is written. Boxes are ticked.
Professional, clear, transparent, safe, SMART even. But not participatory. The evaluator is involved only after the conceptual what-are-we-trying -to-do? why? How? stuff is done. The community, subjects (hate the word) of the intervention, are not involved at all. Everyone almost knows what they’re going to get before they’ve got it. Ho Hum. Is there an alternative? Could there be participatory relationship based evaluation? An evaluation that’s not predefined but plays out in real time as it runs? Something that answers surprising questions in unexpected ways? Something a bit risky? …Maybe
My next evaluation is with a small NGO. I evaluated with them 5 years ago and made some suggestions including a different conceptual approach. They involved me in their outcome-based redesign and I have been mentoring since. Mentoring? Its the official word for sitting on a rooftop with mountains in the background, listening to stories of change and trying to make sense of them together. Somewhere along the way there was a team-building workshop- the team and me trying to define ourselves and how we work. Anyway, I have a relationship with a project that I am almost part of and a team whom I know well. There’s no way I’m an objective professional evaluator.
Nevertheless both they and I were keen to do this together even before the ToR were written. (They had a normal evaluation a couple of years ago. It was- er- normal- answered predefined questions in pre-defined ways, ticked evaluation boxes but missed much of what they do and how they do it. It did not help much). I said how I’d like to evaluate- outcome harvesting- before being told. The project co-ordinator and I decided we’d like the team to actually do the evaluating. But that’s not objective. I discussed that with the funder, sending this explanation of why objectivity is not an issue for me. They agreed, saying learning and depth is much more important then objectivity. Great, I thought. There was still no ToR.
So last week the project co-ordinator in his office in a village, me in a town in India and a funder in a city in Australia all got together on Skype. We started getting to know each other, told each other who we are and how we think. Kind of as if we are all part of a team. We talked about this evaluation, and then Peter in Australia said “What do you two think of asking the community what their evaluation questions are?” Way Cool, I thought. There is still no ToR.
So here we are, this week, with an evaluation team and dates but no ToR. We are working out how to ask “the community” what their evaluation questions are. Who is “the community” anyway,, and what if they have never heard “mulyankan” (evaluation)? What if they ask for a whole lot of stuff like how many new toilets have been built, how many kilogrammes of produce were harvested and how many trainings have been done? Its likely they will ask for this kind of thing which is almost meaningless to me in terms of system change. It is also likely we will get lots of highly specific but not very useful questions if we just ask “the community” what they want to know. Kya Karna? (what to do?)
But we are a team- the project, the funder and me, now with the community thrown in. We have some ideas on who community is. We have some ideas about how to listen to them and create questions whose answers are meaningful to them, to the funder and to the project. We have lots of ideas…. and sometime before the evaluation we might even have ToR. We might not.
Will this crazy process worK? I don’t know. I don’t really know what “work” means for this evaluation. We wrapped up funder, project and evaluator all together, called ourselves a team and put in ideas like participation, understanding and sustainability too. We’re giving that messy bundle to a community. They’ll unwrap it, repackage it in whatever way suits them and give it back to us. Well figure out what to do with it. Is this crazy process worthwhile? Yes! Risky. Edgy. Not normal. Not SMART. But participatory.
Bring it on.