Catching a glimpse of rigorous evaluation practice: Catherine Collingwood

At the Evaluation Re-visited Conference I was blown away by Patricia Rogers’ revelation that the linear cause and effect matrices that have long been taken to describe the theory on which social intervention programmes are based and against which they must be evaluated, are merely a representation of the theory. The matrix is not the theory itself.

As one participant put it “the theory is inside the arrows”. The theory is not self evidently present in the crisp statements contained in the boxes at different levels of objectives within the planning matrix. The theory remains invisible until there has been a social process of working with people to learn together about their reasons for making this choice over that one in the beginning, and this or that adjustment along the way.

It is in the process of understanding together the choices made, that the change theory generating the programme is revealed.  And so a rigorous evaluation practice seeks to evaluate initiatives on their own terms. It means trying to understand intentions and not only plans, choices and not only outputs.

At the moment in this field we call social evaluation, there are no visual representations of dynamically unfolding programme design. Neither do current representations include the ‘sense making’ processes along the journey of the programme’s intervention.

To my mind, striving to make a visual diagram that more fully represents an integration of the planning, unfolding and sense making in each programme intervention, would be rigour in practice.

On a slightly less serious note (although I can tell you it was very serious when we were watching Bafana, Bafana, Cameroon and Ghana!), here in South Africa we have had plenty of opportunity to consider the notions of evaluation, practice and rigour while watching the Soccer (apologies if you call it ‘football’) World Cup. Many questions came to me, such as: How could you tell if a team has a rigorous practice or not? How would you go about evaluating whether a team is successful or not? Before agreeing to evaluate a soccer team, would you have to know all the in’s and out’s of the game? Does losing mean failure? Was it always the best team that won? Is there such a thing as ‘soccer best practice’ to be used as a soccer evaluation template? Should each team be evaluated against their original game plan – or how well they adapted to the conditions that they met along the way… and so on!!!

And finally, a last soccer question: ‘What is the role of the ball in soccer?’

And one of my favourite quotes:

“…try to love the questions themselves as if they were locked rooms or books written in a very foreign language. Don’t search for the answers, which could not be given to you now, because you would not be able to live them. And the point is, to live everything. Live the questions now. Perhaps then, someday far in the future, you will gradually, without even noticing it, live your way into the answer.” (Rainer Maria Rilke: letters to a young poet)

Catherine Collingwood, evaluation consultant