Review of Methodological Approaches to the Study of the Impact of the Microenterprise Credit Programs
Gaile, Gary L
MetadataShow full item record
Purpose and Scope The purpose of this desk study is to review some of the more rigorous assessments of the impact of microenterprise credit programs in order to inform the design of core impact assessments to be conducted by USAID's Assessing the Impact of Microenterprise Services (AIMS) Project. The review covers eleven studies carried out in Asia, Africa and Latin America. It focuses on sample design and execution, temporal issues, analytical techniques, and control methods for eliminating alternative explanations for changes. After discussing the ways in which previous studies have addressed each topic, the authors provide recommendations for the AIMS impact assessments. Description of the Studies Methodology was not the principal goal of most of the studies selected for review, yet, in each, considerable effort and innovation was devoted to methodological issues. Most of these studies were undertaken explicitly to evaluate one or more microenterprise programs. A wide range of variables were covered in the 11 studies. (These are listed in Annex 1.) A few studies centered on a limited number of impact variables, while others looked for impacts at both the household and enterprise levels. While all studies employed quantitative measures and techniques, a couple of them also used qualitative methods. Most were based on data that were collected more than once, but all within a 24 month timeframe. Almost all of them involved a comparison group. Sample Design and Execution Because of the issues of fungibility and selectivity bias, sample design and execution in microenterprise impact studies is complex and critical. Selection bias arises both in terms of the program clients and the location of the programs. The fungibility issue concerns the fact that financial and other resources, including credit and/or the profits from a microenterprise activity, may move between and among various household activities, making it difficult to track impacts. The review of studies shows that there is consensus that some form of quasi-experimental design is appropriate (assuming that an experimental design is not an option). Recommendations include: • sampling should occur with control groups from within program sites and a control population chosen from matched non-program sites; • statistically-equated control groups may be used for individual controls; • in program sites, eligible non-borrowers should be used as control groups; and • an overall sample size of about 500 should allow for effective use of control variables and for dealing with problems associated with longitudinal analysis. Temporal Issues Impact studies are very sensitive to temporal issues. The point at which impacts first begin to occur, and the length of time that impacts are sustained (as well as the rate of change) are subjects of debate. There is consensus that longitudinal analysis is required. Recommendations include: WPDATA\REPORTS\3175\3175-011.w61 (2/97) vi • the research design should include a longitudinal study with an 18-24 month interval between data collection rounds ; • recall methods can be used to enhance the longitudinal profile; • seasonality should be a consideration in research design; • in-depth interviews may reveal "time lines" of credit impacts; and • there may be neglect of long term credit impacts. Analytical Techniques Quasi-experimental design coupled with multivariate statistical analyses are the predominant analytical techniques used in the studies reviewed. Econometricians have used these techniques as complements to econometric modeling. Econometric modeling has the advantage that it is readily generalizable, but also the disadvantages that rigorous assumptions, are required that cannot always be met, and such modeling has a restricted audience. Recommendations include: • multivariate techniques can control for selection bias and endogeneity issues; • choice of techniques should be a function of the type of data collected and their distributional characteristics; • an expanded list of variables should be covered, including social, contextual and locational variables; and, • data cleaning and checks on data validity should be part of the research design. Control Methods None of the studies reviewed successfully controlled for the fungibility of resources between household and enterprise. Selection bias also presents control complications. Linked with both sampling design and analytical techniques, recommendations on control methods include: • statistically-equated control methods are sufficient to address most control issues; • gender is a critical control variable; • continued efforts to control for fungibility must be made; and • control methods should be a function of the data available. LOCATIONAL CONSIDERATIONS Location is given minimal consideration in most impact studies, yet it plays a major role. The location of the program is a major determinant of success. The relative location of clients is likely to be important. Locational changes (e.g., road improvement) also have an impact on program performance. Finally, use of carefully paired, non-program locales as a control method will significantly improve methodological rigor. OTHER ISSUES Too little attention has been paid to alternative methodologies, such as qualitative methods and counterfactual analysis. Similarly, such questionnaire concerns as survey fatigue and the need for back translation have received scant notice. Also, concern for issues related to gathering information WPDATA\REPORTS\3175\3175-011.w61 (2/97) vii in the field is rarely expressed in the studies. Finally, issues such as politics, favoritism, corruption, accountability, and leakages are rarely part of impact research design. CONCLUSIONS Several issues complicate selection of an appropriate methodology for studying the impacts of microenterprise program credit. Two predominate. The first is the issue of fungibility, since credit and other resources may be used for both enterprise and household purposes. The second is the issue of selectivity bias, since both the borrower and the lender "select" participation, which means that loan recipients are decidedly non-random. Debate surrounds program evaluation methodology. Both quantitative and qualitative methodologies have been used, both have positive and negative aspects, and both have achieved acceptance. Valid evaluations can be achieved through a variety of approaches. The papers reviewed for this study indicate that significant "norming" has occurred in the field of microenterprise program impact research on many issues, such as reducing selection bias and improving controls. Some issues, such as fungibility, remain problematic. Methodology "drives" some studies. More care must be taken to fully specify study objectives and to allow these objectives to dictate the types of data that are collected and the methodology that is used.