Also:
Meet a Woman from Bundelkhand | Check-Dams and Irrigation |
Check-Dam Evaluation Study | Bundelkhand Region |
DA Activities in Bundelkhand

Check-Dam Evaluation Study

Over the last nine years Development Alternatives (DA) has facilitated the construction of almost 100 check-dams in the Bundelkhand Region (MP). In 1998 / 1999 DA undertook a study to assess the benefits and appropriateness of check-dams, identify local problems, and generate baseline measures for future interventions. The study also field tested a conceptual and methodological framework for monitoring and evaluating DA sustainable livelihood projects.

It is this monitoring and evaluation process which is being presented here. By sharing the research process we hope to generate discussion on methods of evaluating and measuring sustainable livelihood generation.

The evaluation framework and methodologies outlined in the present study are not intended to be applied verbatim to other sustainable livelihood projects, but rather should be adapted to local conditions, integrating the needs of the communities, project staff and donors.

Evaluation Approach

The check-dam projects are being evaluated within the context of a sustainable livelihoods approach. The framework for the current evaluation study is partly based on a retrospective reconstruction of original project objectives and partly on the framework for analysis of sustainable rural livelihoods set forth by the Institute of Development Studies (IDS), Sussex [Scoones, 1998].

Major Research Questions

The major research issues that guided the evaluation process are:

• expected and potential outcomes of the check-dams as an example of a rural sustainable livelihood intervention
• current development concerns and issues in the villages.
• the nexus between population and environment.

More specifically the questions are:

1. In participating communities, have the check-dam projects:
1. Created more employment?
• altered cropping patterns (# of crops sown and days worked by landowners)?
• altered agricultural-based activities (# of working days for landless laborers)?
2. Reduced poverty through income or production?
   • altered cropping patterns and agricultural-based activities (food versus cash crops)?
   • increased income levels?
   • increased agricultural productivity?
   • had benefits that accrued equally by gender?
   • affected rural-rural and rural-urban migration patterns?
   • increased food self-sufficiency?
3. Increased quality of life?
   • influenced fertility preferences and behavior?
   • had any impact on morbidity and mortality levels?
   • increased families' utilization of the health care system?
   • improved families' nutritional intake?
   • improved the socio-economic status of women?
   • affected interactions among groups and individuals (created conflicts/partnerships)?
   • resulted in equitable distribution of water?
   • had any unintended secondary effects, either good or bad?
4. Increased livelihood adaptation and resilience to stress?
   • had any effect on food security?
   • affected rural-rural and rural-urban migration patterns?
5. Increased natural resource sustainability?
• changed the groundwater and well water levels?
• affected land degradation and soil quality?
• increased vegetation, especially pasture area and fodder available for livestock?
2. What are the major community problems according to the villagers:
   • what priority do the villagers give to the problems they identify?
   • do perceived problems vary by demographic characteristics (sex, age, caste)?
   • in what ways could DA assist these villages in the future?
3. To what extent is population growth:
   • affecting the demand for food and land?
   • affecting the degradation of natural resources and rate of deforestation?
   • in conjunction with prevailing patterns of consumption and production, affecting levels of air, water and land pollution?

Developing Indicators

Indicators to measure progress towards stated goals and objectives are an essential component of any evaluation study. Indicators can:

• be applied at different scales (individual, household, community, region/state, national);
• range from precise objective measures (quantitative indicators) to broad subjective measures (qualitative indicators);
• be generated by donors, project staff or the communities themselves.

In the study, development of indicators was complicated by the fact that project objectives and expected outputs were not clearly outlined or recorded in the project-planning phase. Little written project documentation now exists. Ideally, original project objectives would be linked to previously defined project indicators and expected outputs.

Many of the study indicators for this evaluation based on the reconstruction and interpretation of original project goals and objectives.

This study employs indicators at the individual, household and community levels. While quantitative indicators were used to measure things such as village demographics, qualitative indicators were developed to capture quality of life issues. "Externally", or project staff-defined, impact indicators for the check-dam projects are complemented by "internally", or community-defined, impact indicators of the project which emerged during the course of the primary data collection. Donor-generated indicators are notably absent as this study is being carried out at the behest of project (DA) staff alone.

Methodology

Research Design

The evaluation study employs a quasi-experimental case-control design to compare villages participating in Development Alternative's check-dam projects with socio-economically and environmentally similar comparison communities in the same district that are not participating in the check-dam projects. This activity will also serve as a baseline study in the comparison communities. In addition to secondary data collection, primary data collection was carried out using:

• Key informant interviews;
• Focus group discussions; and
• PRA activities.

These findings are supported and enhanced by secondary data gathered from various government and non-governmental sources.

Sample

A total of nine villages were included in the evaluation study. These were not randomly selected but instead chosen on the basis of three sets of criteria:

Site selection 1: 3 new check-dam sites in the district of Tikamgarh, MP in the Bundelkhand region were selected because they:

• have been operational since 1995 or later
• are part of either the DA watershed development programme or Swajal Project
• are more accessible than other check-dam sites

Site selection 2: 3 older check-dam sites in Datia District, MP in the Bundelkhand region bordering Jhansi, UP were selected because they:

• have been operational for more than 3 years (1994 or earlier)
• are more accessible than other check-dam sites

Site selection 3: 2 socio-economically and environmentally similar comparison communities that are targeted for inclusion in DA's Watershed Development Programme.

In addition, within each village, cluster sampling was used to recruit participants representative of existing caste structures and of both sexes for the focus group discussions.

Procedure

A research team of four native Hindi speakers, three men and one woman, conducted the primary and secondary research with the assistance of two study coordinators. The primary field research was carried out between the months of July and October of 1998. Several delays were encountered due to problems with transportation during the monsoon months and other staff-related setbacks.

The methods and sources used to collect the primary and secondary data for the evaluation study were:

Regional Key informant interviews

Males and females outside the target communities with knowledge of either the agricultural, economic, socio-demographic and/or health situation of the area:

• Dai (traditional midwife)
• ANM (government nurse-midwife)
• Lekhpal officials (revenue officials)
• Local soil and water conservation scientist (Datia)
• School teacher
• Village Key Informant Interview/Village Profile: One respected and knowledgeable male or female within each community

Participatory Rural Appraisal (PRA) Activities

• Groups of 6-15 adults (18 years and older) of mixed sex and caste
• Activities included:
• resource mapping
• social mapping
• women's time use analysis
• gender analysis
• household expenditures chart
• pair ranking of community problems (some separate by sex)


• Focus group discussions using semi-structured interview guide:
• Cluster sampling to obtain four groups of 6-12 adults (18 years and older) for each village: upper caste females; lower caste females; upper caste males; and lower caste males.

Analytical Techniques

Content analysis was performed for the qualitative findings from the focus group discussions conducted in each of the villages. Qualitative data from various PRA activities was also compiled and analyzed thematically.

Quantitative data gathered through PRA activities and household surveys were analyzed using Excel. Additional quantitative information compiled from various secondary data sources was also examined.

Conclusions: Evaluating Sustainable Livelihoods

The field test of this conceptual and methodological framework for evaluating Development Alternatives' sustainable livelihood revealed:

• Baseline data would have greatly enhanced the measurement of sustainable livelihood effects of the check-dam projects in the present study.
• Some PRA activities, such as resource and social mapping, can be carried out with mixed groups of men and women and women will still participate. Other PRA activities, such as the problem (pair) ranking exercise conducted in the present study, are better conducted in single-sex groups so that women's perspectives are not lost.
• The transcriptions of the focus group discussions reveal that they were treated largely as a group interview rather than a relatively free-flowing discussion and exchange of ideas.
• Additional training of DA/Jhansi field staff in focus group moderation techniques, particularly with respect to sensitive issues such as family planning, alcoholism and domestic violence, would be useful.
• Examination of changes in shelter due to increases in income would be a useful indicator in future studies of sustainable livelihoods projects.
• The focus group discussions did not adequately address current behavior pertaining to out-migration or any possible impact of the check-dam on this behavior. The strategy for discussing this issue should have been better clarified at the outset of the study. In future, using a PRA activity such as a mobility map might help to focus the discussion.
• Although alcoholism and domestic violence are thought to be social problems in the study villages, these issues did not emerge in the discussions of marriage, the dowry system or household income and expenditures. The presence of these social ills within a family would be relevant to any sustainable livelihoods analysis, particularly how extra household income is spent. This is a sensitive topic and strategies for examining it should be explored and tested.
• It would be useful to conduct a small-scale household survey (for example, select 15 representative households from each village, or 5-10% of the total households stratified by appropriate characteristics - by cluster or landowning status) to further document, in a more quantifiable fashion, the benefits of the check-dams.
• The conceptual framework and associated indicators used in the present evaluation study should be discussed and debated by DA staff members and a consensus reached on their suitability for use with DA's unique approach to sustainable livelihoods.
• Further field testing of the evaluation methodologies used in the present study is needed with other DA sustainable livelihoods projects.

For more information on the topical results of this study (i.e. the findings regarding the dams as an intervention) please email tara@sdalt.ernet.in.

Also:
Meet a Woman from Bundelkhand | Check-Dams and Irrigation |
Check-Dam Evaluation Study | Bundelkhand Region |
DA Activities in Bundelkhand