Skip to content

Catching a glimpse of rigorous evaluation practice: Catherine Collingwood

July 20, 2010

At the Evaluation Re-visited Conference I was blown away by Patricia Rogers’ revelation that the linear cause and effect matrices that have long been taken to describe the theory on which social intervention programmes are based and against which they must be evaluated, are merely a representation of the theory. The matrix is not the theory itself.

As one participant put it “the theory is inside the arrows”. The theory is not self evidently present in the crisp statements contained in the boxes at different levels of objectives within the planning matrix. The theory remains invisible until there has been a social process of working with people to learn together about their reasons for making this choice over that one in the beginning, and this or that adjustment along the way.

It is in the process of understanding together the choices made, that the change theory generating the programme is revealed.  And so a rigorous evaluation practice seeks to evaluate initiatives on their own terms. It means trying to understand intentions and not only plans, choices and not only outputs.

At the moment in this field we call social evaluation, there are no visual representations of dynamically unfolding programme design. Neither do current representations include the ‘sense making’ processes along the journey of the programme’s intervention.

To my mind, striving to make a visual diagram that more fully represents an integration of the planning, unfolding and sense making in each programme intervention, would be rigour in practice.

On a slightly less serious note (although I can tell you it was very serious when we were watching Bafana, Bafana, Cameroon and Ghana!), here in South Africa we have had plenty of opportunity to consider the notions of evaluation, practice and rigour while watching the Soccer (apologies if you call it ‘football’) World Cup. Many questions came to me, such as: How could you tell if a team has a rigorous practice or not? How would you go about evaluating whether a team is successful or not? Before agreeing to evaluate a soccer team, would you have to know all the in’s and out’s of the game? Does losing mean failure? Was it always the best team that won? Is there such a thing as ‘soccer best practice’ to be used as a soccer evaluation template? Should each team be evaluated against their original game plan – or how well they adapted to the conditions that they met along the way… and so on!!!

And finally, a last soccer question: ‘What is the role of the ball in soccer?’

And one of my favourite quotes:

“…try to love the questions themselves as if they were locked rooms or books written in a very foreign language. Don’t search for the answers, which could not be given to you now, because you would not be able to live them. And the point is, to live everything. Live the questions now. Perhaps then, someday far in the future, you will gradually, without even noticing it, live your way into the answer.” (Rainer Maria Rilke: letters to a young poet)

Catherine Collingwood, evaluation consultant

One Comment leave one →
  1. July 22, 2010 11:21

    Hi, from one South African Catherine to another…
    I find your blogpost really lovely. There are so many things I’d like to respond to. But one point I want to reflect on is your one about visualisation and what is made explicit in our forms of representation or not. I hadn’t really picked that up from Patricia’s presentation but it is something I have pondered over for many years.

    Language (and particularly written language) is such a dominant mode of representation. And because educational systems (in the West/North), particularly, value written language and explicitness so highly, so many of us have come to think that we can make everything explicit and can put it into boxes (which of course is also visual). And that by doing this we capture and represent the world. Of course this is a fallacy. But more so, in this, I believe we lose the ability to think relationally and to think holistically. We end up thinking in terms of parts rather than wholes. One of my favourite academic papers is one by Marilyn Strathern and it is called “The tyranny of transparency”, and it grapples with how the representations of things peel away from the realities they are supposed to represent.

    You noted that someone made the point about “arrows” being where the substance really lies, but in my view, arrows can also be misleading as they carry the idea of directionality and causality so strongly! It is “what happens in between the boxes” that is important, but finding good ways to represent that will always be a challenge.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: