| dc.description.abstract | Purpose and Scope
The purpose of this desk study is to review some of the more rigorous assessments of the impact of
microenterprise credit programs in order to inform the design of core impact assessments to be
conducted by USAID's Assessing the Impact of Microenterprise Services (AIMS) Project. The
review covers eleven studies carried out in Asia, Africa and Latin America. It focuses on sample
design and execution, temporal issues, analytical techniques, and control methods for eliminating
alternative explanations for changes. After discussing the ways in which previous studies have
addressed each topic, the authors provide recommendations for the AIMS impact assessments.
Description of the Studies
Methodology was not the principal goal of most of the studies selected for review, yet, in each,
considerable effort and innovation was devoted to methodological issues. Most of these studies were
undertaken explicitly to evaluate one or more microenterprise programs. A wide range of variables
were covered in the 11 studies. (These are listed in Annex 1.) A few studies centered on a limited
number of impact variables, while others looked for impacts at both the household and enterprise
levels. While all studies employed quantitative measures and techniques, a couple of them also used
qualitative methods. Most were based on data that were collected more than once, but all within a
24 month timeframe. Almost all of them involved a comparison group.
Sample Design and Execution
Because of the issues of fungibility and selectivity bias, sample design and execution in
microenterprise impact studies is complex and critical. Selection bias arises both in terms of the
program clients and the location of the programs. The fungibility issue concerns the fact that financial
and other resources, including credit and/or the profits from a microenterprise activity, may move
between and among various household activities, making it difficult to track impacts. The review of
studies shows that there is consensus that some form of quasi-experimental design is appropriate
(assuming that an experimental design is not an option). Recommendations include:
• sampling should occur with control groups from within program sites and a control
population chosen from matched non-program sites;
• statistically-equated control groups may be used for individual controls;
• in program sites, eligible non-borrowers should be used as control groups; and
• an overall sample size of about 500 should allow for effective use of control variables
and for dealing with problems associated with longitudinal analysis.
Temporal Issues
Impact studies are very sensitive to temporal issues. The point at which impacts first begin to occur,
and the length of time that impacts are sustained (as well as the rate of change) are subjects of debate.
There is consensus that longitudinal analysis is required. Recommendations include:
WPDATA\REPORTS\3175\3175-011.w61
(2/97) vi
• the research design should include a longitudinal study with an 18-24 month interval
between data collection rounds ;
• recall methods can be used to enhance the longitudinal profile;
• seasonality should be a consideration in research design;
• in-depth interviews may reveal "time lines" of credit impacts; and
• there may be neglect of long term credit impacts.
Analytical Techniques
Quasi-experimental design coupled with multivariate statistical analyses are the predominant analytical
techniques used in the studies reviewed. Econometricians have used these techniques as complements
to econometric modeling. Econometric modeling has the advantage that it is readily generalizable,
but also the disadvantages that rigorous assumptions, are required that cannot always be met, and
such modeling has a restricted audience. Recommendations include:
• multivariate techniques can control for selection bias and endogeneity issues;
• choice of techniques should be a function of the type of data collected and their
distributional characteristics;
• an expanded list of variables should be covered, including social, contextual and
locational variables; and,
• data cleaning and checks on data validity should be part of the research design.
Control Methods
None of the studies reviewed successfully controlled for the fungibility of resources between
household and enterprise. Selection bias also presents control complications. Linked with both
sampling design and analytical techniques, recommendations on control methods include:
• statistically-equated control methods are sufficient to address most control issues;
• gender is a critical control variable;
• continued efforts to control for fungibility must be made; and
• control methods should be a function of the data available.
LOCATIONAL CONSIDERATIONS
Location is given minimal consideration in most impact studies, yet it plays a major role. The location
of the program is a major determinant of success. The relative location of clients is likely to be
important. Locational changes (e.g., road improvement) also have an impact on program
performance. Finally, use of carefully paired, non-program locales as a control method will
significantly improve methodological rigor.
OTHER ISSUES
Too little attention has been paid to alternative methodologies, such as qualitative methods and
counterfactual analysis. Similarly, such questionnaire concerns as survey fatigue and the need for
back translation have received scant notice. Also, concern for issues related to gathering information
WPDATA\REPORTS\3175\3175-011.w61
(2/97) vii
in the field is rarely expressed in the studies. Finally, issues such as politics, favoritism, corruption,
accountability, and leakages are rarely part of impact research design.
CONCLUSIONS
Several issues complicate selection of an appropriate methodology for studying the impacts of
microenterprise program credit. Two predominate. The first is the issue of fungibility, since credit
and other resources may be used for both enterprise and household purposes. The second is the issue
of selectivity bias, since both the borrower and the lender "select" participation, which means that
loan recipients are decidedly non-random.
Debate surrounds program evaluation methodology. Both quantitative and qualitative methodologies
have been used, both have positive and negative aspects, and both have achieved acceptance. Valid
evaluations can be achieved through a variety of approaches.
The papers reviewed for this study indicate that significant "norming" has occurred in the field of
microenterprise program impact research on many issues, such as reducing selection bias and
improving controls. Some issues, such as fungibility, remain problematic. Methodology "drives"
some studies. More care must be taken to fully specify study objectives and to allow these objectives
to dictate the types of data that are collected and the methodology that is used. | en_US |
