Category Archives: monitoring and complexity

Three Considerations when Measuring “Success” in Development Cooperation: A Conversation with Zenebe Uraguchi

This post was written by Zenebe Uraguchi and originally published on the Helvetas Inclusive Systems Blog. It is reposted here with permission.

Two years ago, Zenebe Uraguchi of Helvetas had a conversation with Rubaiyath Sarwar of Innovision Consulting on how fixation on chasing targets leads programmes in development cooperation to miss out on contributing to long-term and large-scale changes. In March 2019, Zenebe met Marcus Jenal in Moldova. Marcus thinks a lot about how complexity thinking can improve development.

This blog is a summary of their dialogue on three thoughts that development practitioners who apply a systemic approach need to consider when measuring success in terms of contributing to systemic change.

By systemic change, we mean changes in the dynamic structures of a system – rules, norms, customs, habits, relationships, sensemaking mechanisms or more generally: institutions and world views – that shape the behaviours or practices of people – individuals, businesses, public sector agencies, civil society organisations, etc.

ZU: Programmes that apply a systemic approach in development cooperation often struggle to measure the systemic change they effect. A couple of years ago, Michael Kleinman wrote in The Guardian, arguing that, “the obsession with measuring impact is paralysing [development practitioners].” Without being obsessed with measurement, I believe development programmes will require a measurement system that’s right-sized and appropriate in scope and timeframe for effectively measuring impacts.

MJ: For me, the challenge is how to find a broader way of effectively showing successes when attempting to stimulate systemic changes. This means not reducing our success factors to the measures that we know how to measure. We need to keep an eye on how our role is contributing to broader change, for example, by using different methodologies and appreciating the perspectives these provide us with. This’ll, for sure, help in demonstrating how a programme contributed to meaningful change in a system. A programme will need to weave different sources and types of evidence into a coherent story. Of course, we need to also make it clear in the story we tell that there’re other factors that influence the change programmes have contributed to.

ZU: In my recent reflection about the Market Systems Symposium in Cape Town, I emphasised the concern that the evidence on impact of programmes that apply a systemic approach is thin. Among others, one of the key challenges is the tension between short-term and long-term results. Can such a tension be managed or reconciled?

MJ: This tension exists in most programmes that apply a systemic approach. On the one hand, there’s a requirement for showing results within a given time frame (e.g. partners have taken new ways of working and showing successes in terms of investment and job creation). This often requires programmes to use incremental interventions with no transformational effect. On the other hand, programmes will also need to invest in more fundamental, long-term systemic changes (e.g. changes in how different institutions interact, improved participation in labour markets).

The key point here is whenever we design interventions or prepare sector strategies, we need to pay attention to explaining how we expect changes to happen and in what sequence. In other words, we need to explicitly state which changes we expect in the short term, in the medium term and in the long term. By categorising the effects of our interventions in this way, I think it’s possible to come up with different types of indicators appropriate for the different stages. Indeed, in using such ways of measuring changes, programmes should work with donors and their Head Offices to manage expectations and tell the narrative on how they expect changes to happen over time.   

ZU: Many development programmes operate in complex and dynamic contexts. I’m aware that adaptive management can sometimes be viewed as an excuse for “making things up as programmes go along”. Yet, the point I’m trying to make is that the context can shift quickly, and strategies need to be adapted continuously. This means that monitoring and results measurement needs to adapt to such changes. For example, having access to reliable and timely information through an agile monitoring and results measurement system is crucial.

MJ: I agree with your point. Development practitioners are still evaluating programmes that work towards facilitating systemic changes by using methods that aren’t adjusted to the shift to a more systemic way of doing development. The evaluation methods are following a “traditional” model adapted to show direct effects (clear discernible cause-effect links, linear effects, lack of feedback loops). For me, this’s to a certain extent unfair towards programmes that take a systemic approach. So, we need to ask ourselves two questions: “what success means” and “how we measure success accordingly” for programmes that work towards systemic change. Only then it’s reasonably possible to show whether an initiative has been successful or not. An immediate follow-up question then needs to be: how can this be done? There’re good examples of methodologies that’re able to capture systemic effects in the evolution community and to a certain extent also in the social sciences.           

ZU: Systemic approaches aren’t entirely new. The approach puts lessons from decades of development work into a set of principles and frameworks to guide development programmes in their design, implementation and results measurement. If this is the case, then why’re we still struggling to figure out how to effectively measure success (on a system level) in development cooperation? Or is it the case that “development isn’t a science and cannot be measured”?

MJ: As I said above, perhaps it isn’t due to the lack of ability to show these changes but the adoption of the appropriate methods in our field. Oftentimes we start development initiatives with a good intention to change or improve a system. We’re then soon confronted with the question: “how are we going to measure such a change?” As we naturally default to the good practices and standards that are used in our field (or are even forced to use them by our donor), which’re still predominantly based on a linear logic, we’re automatically only measuring the effects that can be captured by such methods: direct effects. This, in turn, again affects the way we design our interventions, or the way we work with partners to stimulate systemic change.

It’s a circular logic, you see: our focus will be on achieving targets defined through measures that we intend to use to measure success with – and if these measures aren’t systemic, our focus will not be on systemic change. This’s what I call “the self-fulfilling prophecy” of measuring changes in development cooperation. Let me provide you an example:

ZU: Great points, Marcus. So, what do we make of our conversation? I see three key messages regarding measurement of success: first, the measures we choose will define or at least influence the way how we work, second, we need to choose the right ways of measuring success that’s in line with the kind of approach we use, and third, the importance of learning from recent experiences of evaluating success in the wider evaluation community.

MJ: That’s a good summary. Let me explain these three takeaways a bit more.  

Want to measure systemic change? Here’s a refined complexity sensitive framework.

Systemic change has become a catch phrase in recent years, not only in the field of Market Systems Development. I have blogged about it before (for example here and here). The question I want to address in this post is how we can conceptualise systemic change as a first step in developing ‘Theories of Systemic Change’ and evaluating systemic change initiatives. And all of this in the face of complexity and unpredictability of how complex systems change. Continue reading

An alternative to a Theory of Change approach

I have been blogging quite extensively about the Theory of Change (ToC) approach in recent months. My blog posts reflect a process that I have been going through as part of my different work engagements: adapt ToC approaches to be more sensitive to the complexities development programmes face in their day-to-day work.

Systemic Insight emphases.jpg

Different phases of Systemic Insight

In parallel, with my colleagues at Mesopartner we keep doing research on understanding complex realities and our human reaction to them based on cognitive science, understanding the process of economic change, making decisions under conditions of uncertainty, and managing highly resilient organisations. In these contexts, ToC has limited applicability and a number of drawbacks. Therefore, we have been working on an alternative approach to the ToC approach which we built from the ground up based on our growing understanding of how complex systems work and how involved actors can lead a process of exploration and change. The approach is called Systemic Insight. Continue reading

Refining the Complexity aware Theory of Change

When I wrote my last post about experimenting with new structures for a complexity aware Theory of Change (ToC) in Myanmar, I had a few elements in place, but still some questions. Going further back to an earlier post, I was clear that differentiating between clear causal links for complicated issues and unpredictable causalities for complex ones is critical. I have been thinking about that a lot and last week I have taught a session on monitoring in complex contexts and I think I have found the final piece of the puzzle. Continue reading

Experimenting with new structures for Theory of Change

Last week I was in Myanmar working with a market
systems development programme. The main task of this trip is to work on the project’s monitoring framework. To set the stage for that, we are working on revising the project’s theory of change (ToC).

messy toc.jpg

A messy theory of change

Theory of Change is a bit of a contentious beast in my set of tools. As I am thinking and writing a lot on complexity and complex systems, I am aware that causality in complex systems can hardly ever be reduced to a straight line between two boxes and it is even more difficult to predict in advance how change will look like. It is not just that causalities are difficult to disentangle or predict in advance (it’s easier using hindsight), but that because of emergence there are other causalities at work than the linear material – billard-ball like – causality we are used to. But this is the topic of another blog post. So for me, Theory of Change is not an instrument to predict what change will happen but to create a coherent picture that explains why the project is doing what it is doing. Continue reading

Measuring transformations in attitudes

Following up on my last blog post on a new framework for systemic change, I would like to present here the main methodology we used to measure whether there have been transformations in the attitudes of farmers. The approach we used was Cognitive Edge’s SenseMaker®, which allowed us to deeply scan for changes in attitudes and beliefs beyond mere observation of changed behaviours. Continue reading

A new framework for assessing systemic change

Over the last year or so I was hired by a large market systems development programme in Bangladesh to develop a new framework for assessing systemic change for them. We did an initial feasibility study and then a larger pilot study. The report of the pilot study has now been published. Rather than to bore you with the whole report, I would like to share the conceptual thinking behind the framework and the framework itself in this post. In a later post, I will share the methodology. This is not the end of all wisdom and the silver bullet framework everybody has been looking for. For me this is an important step to bring my work and thinking over the last couple of years together into something practically applicable. But this work is not done as I am embarking on a longer research project on systemic change. So there is more learning to come and with it more development of this tool. Please share your thoughts, which would help me to further improve the framework. Continue reading

Don’t over-design your ToC

Getting too eager about building the perfect Theory of Change (ToC) for your organisation, programme or project can lead to an over-designed ToC that can be more of a hindrance than a help to manage and learn. It sucks up a lot of time and team resources to build but then gets out-dated extremely quickly. A ToC should be an idea that is alive and dynamic. For me a ToC is more useful if it is a sketch on the back of an envelope after an intense discussion rather than a page in a high-gloss brochure. A ToC in a complex setting is necessarily imperfect. But it can still be extremely useful. Continue reading

ToC – all harmony?

Continuing my little emerging series on Theories of Change, there is another issue that I feel is very important in connection with complexity-informed Theories of Change: they do not need to be based on total agreement among the stakeholders. On the contrary, it is important to understand where there is agreement on causalities among the stakeholders and where there is not as this gives us important insight on the complexity of specific links in the logical chain.

When we look at the Theory of Change literature, participation comes up as an important if not central element of a Theory of Change process. And it undeniably is. Bringing in a wide range of stakeholders ensures that we get all or many of the diverse perspectives reflected in the Theory of Change process – and as I have written earlier, understanding diverse perspectives is a corner stone of systemic thinking. Continue reading

Adjusting a Theory of Change midway

I got a very good feedback on my blog post last week on complexity informed Theories of Change, it was shared widely on Twitter. But I also got some questions. One person pointed out the fact that the method I described in that post was mainly focusing on new programmes that are developing their first Theory of Change. But what about programmes that are in the middle of implementation? Programmes where the programme team sense that things are not going the way they are supposed to according to their Theory of Change, Logframe or project plan. Should the managers of such programmes just stop operations and go back to the drawing board to develop a complexity aware Theory of Change? This is in most cases not possible, unless things are really going badly. How can these programmes incorporate some of the ideas of complexity informed Theories of Change?
Continue reading