Tag Archives: results measurement

Some principles for measuring change in market systems

CGAPThis blog post was originally published on the Website of CGAP as part of a series of blog posts on measuring change in market systems development under the title “New Funding Approaches Call for a New Way of Measuring Impact.” CGAP (the Consultative Group to Assist the Poor) is a global partnership of 34 leading organizations that seek to advance financial inclusion.

The focus of financial sector development is shifting. Development organizations funding financial inclusion now operate with a vision of sustainability, resilience and impact at scale, and their goals stretch beyond building individual institutions. Now, they aim to improve the whole ecosystem for financial services, taking a facilitative rather than a direct intervention approach. Continue reading

Syntesis paper out now: Monitoring and measuring change in market systems

I am really happy to announce the publication of the Synthesis Paper of the so called ‘Systemic M&E’ initiative. The paper is the synthesis of conversations that started in MaFI in June 2010 and a series of online and in-person conversations that took place in the second half of 2012. It captures the voices of practitioners, academics, donors and entrepreneurs who are trying to find better ways to monitor and evaluate the influence of development projects on market systems and learn more, better and faster from their interventions. Continue reading

A bottom up perspective on results measurement

Thanks to my engagement in the ‘Systemic M&E’ initiative of the SEEP Network (where M&E stands for monitoring and evaluation but we really have been mainly looking into monitoring), I have been  discussing quite a bit with practitioners on monitoring and results measurement and how to make monitoring systems more systemic. For me this bottom up perspective is extremely revealing in how conscious these practitioners are about the complexities of the systems they work in and how they intuitively come up with solutions that are in line with what we could propose based on complexity theory and systems thinking. Nevertheless, practitioners are often still strongly entangled in the overly formalistic and data-driven mindset of the results agenda. This mindset is based on a mechanistic view of systems with clear cause-and-effect relationships and a bias for objectively obtained data that is stripped from its context and by that rendered largely meaningless for improving implementation. Continue reading

Systemic monitoring

Wordle

I have been very lucky to work for an initiative termed ‘Systemic M&E’ (although we talk more about the ‘M’ than the ‘E’), focusing on ways to move away from a linear and static understanding of the systems we work in and develop tools and approaches that let us monitor changes in the complexity of the real world. Continue reading

Spotting ’emerging patterns’ to report on changes

In a training on evaluating projects I attended a while ago, a representative of the Swiss Charity HEKS presented their results measurement (RM) system. The presentation caught my immediate attention and interest since HEKS is using principles of complexity theory as a basis for their RM framework. Based on this rather experimental framework, the organization published a first ‘effectiveness report‘ in March 2011. I want to present some of the interesting features of the RM system, based on the effectiveness report.

HEKS acknowledged when building their RM framework that development takes place in complex and dynamic systems with the consequence that the behavior of such systems is largely unpredictable and, thus, effects of interventions also hard to predict.

This challenging perspective implies a different understanding of cause and effect. Connected to their environment, living systems do not react to a single chain of command, but to a web of influences.

As a consequence, HEKS does not base its projects on rigid impact logics and impact chains, but they are conscious that

HEKS cannot always objectively trace the effects of its actions, but can make its intentions, input and observations transparent.

As a consequence, HEKS’ particular approach focuses on the changes observed and experienced by different stakeholders involved at several levels of their projects.

The focus is more on the significance than on the quantification of such changes for the people who experience them. HEKS herewith takes a path different from strict measurement and hard data collection. Its aim is to grasp and understand the changes in the purpose, identity and dynamics that hold and drive the systems it gets involved – rather than to measure their ever changing dimensions.

Subsequently, HEKS’ method is to adopt a bird’s-eye view, look for ‘emerging patterns‘ and try to interpret them. Qualitative data is collected on three levels, i.e., the indivudual level, the project level and the programme level through methods like ‘Most Significant Changes’, monthly newsletters and annual reports focusing on observations of different level staff as well as a two days workshop for compilation.

Nevertheless, HEKS defined 10 key indicators that are collected for all countries they are active in. These indicators are for example number of beneficiaries, income increase, yield increase, etc.

For me, this is a very interesting approach and it resonates very well with the discussion on ‘experiential knowledge and staff observation’ of the GROOVE network that I mentioned in my last post. Also the staff observation have as an implicit goal to grasp emerging patterns of positive changes in the system the project tries to influence in order to amplify this change.

Owen Barder, on whose presentation on evolution and development I wrote in my last post, is asking for more rigorous evaluation of project impacts in order to be able to see what works and what doesn’t. Is the RM framework proposed by HEKS rigorous enough to comply with Owen’s demand? After all, HEKS’ approach is not using result chains at all, although they are one of the mainstays of results measurement – at least according to the DCED Standard on Results Measurement. Are the 10 universal indicators enough? And what about the attribution of the changes and emerging patterns?

When I read through the four patterns that were described by the HEKS effectiveness report, I see that they are very much focused on the community level – naturally, since this is where also the focus of interventions lie. Here is an example:

Pattern 1: Sustainable development starts with the new ways in which people look at themselves. Women especially become a driving force in the development of their communities.

Or another one:

Pattern 2: People who are aware of their rights become players in their own development. They launch their initiatives beyond the scope of HEKS’ projects.

The question that immediately pops up in my mind is: What are the consequences of the projects’ actions on the wider system, beyond the community? What are the ripples that the successful projects have throughout the wider system, e.g. in the market system or the policy environment? Or even more fundamentally: Can we achieve changes in the wider system by focusing on the community level? What additional interventions are needed?

There are still many open questions, but for me, HEKS is making a huge and courageous step in the right direction.