Measuring Transformation

> Posted by Bobbi Gray, Research Director, Freedom from Hunger

[getty src=”107953528?et=N-ti1Dh0TYxzWsk0oHctDQ&sig=ohTg-74FOrnUDD5V5JtetTs7rwUi7ycpCTt4v7oMEM8=&caption=true” width=”594″ height=”419″]

While recent research indicates that access to and use of microcredit alone is not transformative for the average client served (see “Where Credit Is Due”), there has been very little discussion about the types of indicators being used to measure “transformation” in the ongoing debates. In fact, it seems that we all have accepted the general findings that microcredit has only had modest impacts on, along with other indicators of poverty and well-being, education, health, and social capital because the randomized controlled trials (RCTs) have said so. There needs to be greater thought and debate about the choices of indicators used to support these conclusions.

Freedom from Hunger over the past 20-plus years has integrated health with microfinance and helped build a body of knowledge indicating that microfinance plus health services can enhance health outcomes. In an ongoing partnership with the Microcredit Summit Campaign, supported by Johnson and Johnson, we have pilot-tested a series of health indicators that financial service providers (FSPs) can use to track client health outcomes. This pilot test was built on years of experience of evaluating health outcomes with our FSP partners, as well as on similar experiences of developing common tracking indicators in the health sector. We created a list of criteria to assess the types of indicators we felt would be meaningful to track—for individuals with and without health services – which included dimensions of feasibility, usability, and reliability. Initial results have been shared in several webinars with SEEP and the Social Performance Task Force.

It’s important to note that this pilot test effort was not about “proving” impact, but rather developing common techniques for monitoring client outcomes that FSPs could use over time. However, this experience has shown how difficult it is to identify indicators that best measure certain health outcomes. What initially might appear as an intuitive indicator to use—for example, how often a person reports being ill or seeking medical treatment—is found to be more difficult than expected. Morbidity—or reports of illness—is not an easy measure for health sector actors or those who directly work to improve health outcomes because it is influenced by the seasons, by specific efforts, and other factors, so care has to be taken when interpreting results. Reports of seeking medical treatment are complicated by whether people are satisfied with the services they can seek and may not always reflect financial capability but preferences or lack of available health services.

Acknowledging this experience and the importance of RCTs for our work, a discussion is still warranted on the bigger picture behind client outcomes indicators: how do we define “transformation” and which indicators do we use to measure it.

For example, the RCTs found that household access to and use of microcredit by itself led to no transformative effects in relation to health; however, these studies also relied on child morbidity rates, incidence of serious illness in the prior three years, and household spending on health as key outcomes to measure the impact of microcredit on health. On the one hand, one could argue that these were the right indicators because they represent important high-level health outcomes. On the other hand, measuring change in child morbidity rates during a short one- to three-year window may not be the most relevant and useful indicator to use to detect impact for FSPs. Additionally, it is not clear whether seasonality issues were taken into account for this indicator. Changes in household spending on health may also not reveal important impacts if the local health system is not easily accessible or a desirable place from which to seek health care, or if people tend to rely on traditional medicine. Yet, it is also important to note that until more information exists from FSPs collecting data on health, it is unclear which indicators will likely be the most useful for understanding impact (through use of evaluations) and understanding changes in client well-being over time (through use of impact monitoring processes).

Education and women’s empowerment were also assessed in the RCTs. Ratios of school-aged children in school and amounts spent on education were typically used to measure changes in educational outcomes. Women’s empowerment was acknowledged throughout the six RCTs as being tricky to measure, and the studies tended to rely on indices that combined a series of indicators assessing decision-making, mobility, and ownership of assets. In some cases, women’s empowerment was measured even if lending was not exclusive to women. The RCT conducted with Compartamos in Mexico was arguably the most comprehensive in the types of “transformative” indicators assessed, as it included subjective measures such as stress, happiness, and trust. It is unclear whether these are the most important indicators for education and women’s empowerment. Are they the best and most accurate indicators to tell us whether transformation is occurring?

As participants in a financial services sector, in collaboration with our academic colleagues, we have to identify better shared indicators of success. While it might seem easy to choose indicators, this is as much art as it is science. Even with the national Demographic and Health Surveys, debates occur about the best sets of indicators to be used—and these surveys are considered fairly standardized across countries.

There is growing energy around the commitment to identify better common client outcomes. The client outcomes working group of the Social Performance Task Force is one focal point for this energy. The Microfinance CEO Working Group‘s social performance sub-committee is another group of committed individuals who want to make more meaningful comparisons and statements about where we expect to see impact and how we can help FSPs monitor the data themselves. The Microfinance Genome from Mission Measurement is another.

The debate cannot stop with whether or not transformation has occurred. The debate has to include how we define transformation and how we measure it. This means that, along with the much-scrutinized data, we have to pay equal attention to the indicators. If there is no silver bullet for reducing poverty, neither does there appear to be a silver bullet for measuring transformation.

Have you read?

Measuring the Impact of Microcredit – Six New Studies

Reflections on the Latest Studies on Microcredit

Integrating Health and Microfinance in India