Following up on my last blog post on a new framework for systemic change, I would like to present here the main methodology we used to measure whether there have been transformations in the attitudes of farmers. The approach we used was Cognitive Edge’s SenseMaker®, which allowed us to deeply scan for changes in attitudes and beliefs beyond mere observation of changed behaviours.
The SenseMaker data collection process has three steps:
- By asking for a specific experience, the respondents’ memory is triggered to mentally place them in a situation related to the field of interest that they have lived through.
- Respondents record a narrative of that experience and, through so doing, bring the experience into working memory.
- Respondents are asked to signify (interpret) their own narrative using a set of carefully crafted questions.
‘Self-signification’ of the stories by the respondents is a key aspect of this method. It allows respondents to interpret their own experiences, avoiding expert bias. Self-signification furthermore enables respondents to give additional meaning to each situation that is not explicitly expressed in the story. This provides a rich set of quantitative meta-data for analysis.
For this study, respondents were asked to share a specific experience of support they have received from people outside their family related to farming. They were explicitly invited to share both positive and negative experiences. After sharing an experience, which was recorded with a mobile device, the respondents were asked to answer a number of signification questions built up of triads (Figure 1), dyads (Figure 2) and multiple-choice questions.
Respondents are asked to place a dot in the triad to the left in Figure 1, signifying how the story represents the blend of the three aspects in the corners. All the dots of all the stories are then visualised during the analysis as seen in the triad on the right of Figure 1. Colour coding allows differentiating between different characteristics of the respondent (e.g. gender, age bracket, poverty level, etc.). Individual stories remain accessible throughout the analysis to give context to the visualised patterns.
In total, 497 stories were collected from four geographical areas. The characteristics of the sample are shown in Figure 3.
The sample was designed to be large enough for statistical analysis. It was also intended to assess the scale of observed changes. The sample population was divided into three groups: an intervention group, which was about 50% of the sample; a spillover group of farmers from adjacent areas where there is a potential for them to have profited from the interventions indirectly; and a comparison group from areas farther away without any contact to the intervention. The respondents were also asked to tell whether the support they mention in their story has always been available or has only become available in the last year – which is essentially the period in which interventions have been going on. As there is no baseline, this allows for a crude before/after comparison, in addition to the comparison between intervention, spillover and comparison groups.
The analysis of the data was done in a collaborative effort between the two consultants involved and the project team.
The whole report of the study has now been published and can be downloaded here: Jenal 2016 – A New Framework for Assessing Systemic Change