Services: Program Evaluation
Evaluation is a platform for learning, accountability and focused strategy.
Program evaluation is central to organizational learning, performance improvement and accountability. Through evaluation we can determine the efficiency, relevance, effectiveness, sustainability and institutional impacts of development activities. Impact evaluation helps us to better understand the extent to which activities reach the poor and the magnitude of their effects on people's welfare.
SI provides a full range of evaluation services. Our approach is built on international standards of excellence requiring evaluations to be useful, practical, ethical and accurate. We also build the capacity of organizations to monitor and evaluate their own programs in order to better achieve their missions.
Range of Services
- Impact Evaluation-systematic assessment of effects on individual households, institutions, and the environment caused by a given development project, program or policy.
- Evaluations for Program Improvement-evaluations focused on improving projects and programs during the project life.
- Participatory and Empowerment Evaluation-evaluations with central objectives to foster participation, learning and empowerment among local stakeholder groups.
- Evaluation Capacity Development-building organizational skills, incentives and systems to use evaluation in order to achieve results.
- PROJECT details
-
El Salvador-Impact Evaluation Design & Implementation Services for Infrastructure Components of the Millennium Challenge Account (MCA): Millennium Challenge Corporation (MCC), 2007-2012
The MCC El Salvador Compact Program focuses on the Northern Zone of El Salvador, an area that is home to one-half of the country's poorest municipalities and where 53% of the households are poor and 28% experience extreme poverty. The Program seeks to alleviate this situation through increased economic growth via three projects: Human Development (Education and Training and Community Development); Productive Development; and Connectivity. As prime contractor, Social Impact was awarded a five-year contract to design and implement rigorous impact evaluations of the Community Development sub-activity (Water and Sanitation, Rural Electrification and Community Infrastructure) and the Connectivity project (a Transnational Highway and a Network of Connecting Roads). During Phase One, the SI team is working with MCC, MCA-El Salvador and other key stakeholders to develop evaluation methodologies (e.g., randomized control trials and propensity score matching) and an implementation plan. SI is also modifying the annual national household survey and will play a lead role in developing a panel survey, including determining sample size, design, and questions. During Phase Two, the SI team will provide guidance on studies being undertaken and on data collection. During Phase Three, the SI team will implement the evaluation design, provide oversight on data collection, and conduct the analysis. Throughout this whole process, the SI team will work collaboratively with and provide training to local counterparts in order to increase their capacity to design and implement evaluations.
-
Impact Evaluation Services for the Millennium Challenge Corporation, 2007-2011
SI is a lead organization on an Indefinite Delivery, Indefinite Quantity (IDIQ) contract to provide strong impact evaluation services to the Millennium Challenge Corporation (MCC) for their economic growth and poverty reduction programs in developing countries. With subcontractors Research Triangle Institute, Inc (RTI) and International Food Policy Research Institute (IFPRI), our impact evaluations will enable MCC to demonstrate the economic and social return on their investments and provide MCC, host country policy makers and donors with strong evidence for making decisions about how to allocate development resources. SI provides MCC with evaluation approaches using randomized control trials, and in instances where these are not feasible, SI provides the most rigorous quasi-experimental designs as alternatives. SI works closely with and provides capacity building training for the MCC and local counterparts during each phase of the project to ensure the highest quality work and a feeling of shared ownership over the evaluation process and products. SI also promotes the utilization of MCC impact evaluations by national counterparts and develops a learning culture to disseminate and utilize the findings and lessons more broadly. SI was also awarded the first multi-year task order to provide impact evaluation services for MCC's new program in El Salvador.
-
Armenia Media Sector Mid-term Evaluation, USAID/IREX, 2007
The primary mission of the Core Media Support Program of Armenia (CMSPA), a four-year program implemented by IREX and funded by USAID, is to foster self sustainability in the Armenian media sector. The main element of the program is designed to assist commercially viable targeted independent media outlets through the provision of loans. As IREX reached the mid-point of the program, it commissioned SI to conduct a mid-term evaluation of the program to gain an independent assessment of the impact on targeted Armenia media outlets (and the wider media sector) of prior and on-going program initiatives and recommendations for possible future initiatives. The SI team was asked to carefully assess the program and develop a set of practical recommendations for program improvement aimed at maximizing program results during the remaining time of the CMSPA program. SI presented its findings and recommendations through debriefs for IREX field and HQ staff, as well as via a final evaluation report. The team conducted its research through a desk review, content analysis and a field trip to Armenia. IREX is currently implementing these recommendations to improve program performance, especially in the areas of data analysis, reporting and extending the reach of the loan program to additional regions in Armenia.
-
Final Evaluation of Emergency Capacity Building Project, 2007
The Emergency Capacity Building Project (ECB), funded by the Bill and Melinda Gates Foundation, is a cooperative undertaking of seven large humanitarian agencies (CARE, Save, CRS, Oxfam-UK, IRC, Mercy Corps and NETHOPE) aimed at improving their emergency response capacity and in particular the speed, quality and effectiveness in saving lives, improving welfare and protecting the rights of people in emergencies. SI conducted the final evaluation for the project focusing on a review of project design, implementation, effectiveness, results and outcomes while reflecting on processes used to promote partnerships and organizational learning within the humanitarian sector. The evaluation was also planned to inform future design and operations of emergency capacity building projects. Using a mixed method evaluation approach, including extensive interviewing, desk review and an innovative partnership survey, SI supported the ECB team to consolidate an understanding of results and lessons learned from their first phase of work together and to feed this learning directly into the design of Phase II of the project.
-
Colombia - Strengthening SINERGIA, the M&E System of the Colombian Government, 2007-08
SI is working with the Colombian government to improve the rigor, sustainability, and capacity of their government-wide M&E system SINERGIA, recognized as one of the leading M&E systems in Latin America. To this end, SI will review and rate the evaluation methodologies currently employed, and will offer additional techniques to strengthen their evaluation work. SI will develop a system for prioritizing evaluations based on the magnitude of expected effects, the number of people affected and the budget. SI will also review the adequacy of the current incentive system and will provide recommendations on how to promote greater utilization of M&E information by all government entities. During this entire process, SI will work closely with SINERGIA and help build their evaluation capacity to improve public sector performance.
-
Impact Evaluation Services for the Millennium Challenge Corporation, 2007-2011
SI is a lead organization on an Indefinite Delivery, Indefinite Quantity (IDIQ) contract to provide strong impact evaluation services to the Millennium Challenge Corporation (MCC) for their economic growth and poverty reduction programs in developing countries. With subcontractors Research Triangle Institute, Inc (RTI) and International Food Policy Research Institute (IFPRI), our impact evaluations will enable MCC to demonstrate the economic and social return on their investments and provide MCC, host country policy makers and donors with strong evidence for making decisions about how to allocate development resources. SI provides MCC with evaluation approaches using randomized control trials, and in instances where these are not feasible, SI provides the most rigorous quasi-experimental designs as alternatives. SI works closely with and provides capacity building training for the MCC and local counterparts during each phase of the project to ensure the highest quality work and a feeling of shared ownership over the evaluation process and products. SI also promotes the utilization of MCC impact evaluations by national counterparts and develops a learning culture to disseminate and utilize the findings and lessons more broadly. SI was also awarded the first multi-year task order to provide impact evaluation services for MCC's new program in El Salvador.
-
Sri Lanka—Final Evaluation of the Sri Lanka Transition Initiative: USAID/OTI, 2007
The Sri Lanka Transition Initiative (SLTI) was launched in February 2003 to support peace talks between the Government of Sri Lanka and the Liberation Tigers of Tamil Eelam (LTTE) through generating greater support for a negotiated peace settlement to end the longstanding internal conflict. The objectives of this program were to: i) change attitudes sustaining the conflict; and ii) mobilize and link peace constituencies through local-level activities promoting collaborative decision-making. Social Impact conducted a final evaluation of the program that documented accomplishments and lessons learned for USAID. As a result of the document review, extensive interviews and a nearly three-week field mission to Sri Lanka, the evaluation team analyzed the impact of the SLTI program as well as lessons learned and best practices to benefit future programming. The evaluation was used by OTI to develop a strategy for the final phase of the program. The report highlighted that the SLTI substantially contributed to livelihoods, local infrastructure and, in some cases, the quality of life in Sri Lanka by responding to the challenge of the tsunami and post-tsunami rehabilitation period with commitment and ingenuity in the hardest hit areas.
-
Assistance to Russian Orphans 2 Program Evaluation: USAID, 2006
In response to the challenges of Russia's ill-equipped and ineffective child welfare system, the Assistance to Russian Orphans 2 (ARO2) program, sponsored by the USAID and implemented by IREX and its local partner, the National Foundation for the Prevention of Cruelty to Children (NFPCC), aims to increase the use of improved health and child welfare practices through the implementation of new child abandonment prevention and family-based care services. SI conducted the ARO2 Evaluation in the Russian Far East, Siberia and Moscow to develop practical recommendations for program improvement aimed at consolidating program results and to improve program monitoring and evaluation systems so that results can be better documented during the next phase of the program, ARO3. The report thus evaluated the strengths and challenges of ARO2 to institutionalize and consolidate the gains that were made during ARO2 and to ensure that mechanisms are in place for their spread throughout the Russian Federation during ARO3.
-
Liberia—Final Evaluation of USAID/OTI Program, 2006
The Liberia Transition Initiative was launched in February 2004 to help advance an inclusive, peaceful transition in Liberia following 14 years of civil war that ended with the 2003 Comprehensive Peace Agreement. Through this program, OTI worked to restore critical transitional governance and civil society functions; increase the understanding of key political transition issues; and promote community integration and peaceful resolution of conflict. SI performed the final evaluation of this program which included activities that focused on youth and community infrastructure development. Specifically, the evaluation was designed to explore the extent to which OTI's efforts advanced the prospects for an inclusive and peaceful political transition in Liberia. By adopting a context-specific approach that takes into account the needs, interests and indicators for success that are meaningful to the beneficiaries of the program, SI produced an evaluation report that was used by OTI to develop the second phase of the Liberia Transition Initiative. The report highlighted OTI's vital ability to bring immediate evidence of the benefits of peace to a war stressed population, and to reinforce that peace by using grants to foster effective governance and social and political cooperation.
-
Strategy and Impact of the Iraq Transition Initiative: USAID/OTI, 2006
The Iraq Transition Initiative (ITI) was launched in April 2003 to work closely with the Iraqi Interim Government and the United States Embassy in Iraq. The objectives of ITI were to identify and fill crucial gaps in the US Government's assistance efforts at the national and local levels and increase public support for the Interim Government. Activities centered on the disbursement of small grants to local Iraqi groups and institutions. Since the program's beginning, ITI issued approximately 5,000 small grants totaling $335 million. Social Impact conducted a final evaluation of the impact of this project. As a result of the document review and extensive interviews, SI analyzed quantitative and qualitative information that identified the strengths and weakness of the program as well as lessons learned and best practices to benefit future programming. The findings of the evaluation are being used by OTI to demonstrate results of the program and lessons learned are being used to improve OTI-military relations, better align strategic and programmatic objectives and to improve the results-orientation and monitoring of future OTI projects in transition countries.
-
Sudan—Final Evaluation: The Sudan Peace Fund and the South Sudan Transition Initiative Program: USAID/Office of Transition Initiatives (OTI), 2006
The 2005 Comprehensive Peace Agreement (CPA) signed between the Government of Sudan and the Sudanese People's Liberation Movement/Army presented an historic opportunity to end 50 years of fighting between the North and the South. Peace, however, was contingent upon internal Southern stability, which had been undermined by 23 years of inter-ethnic fighting. For three years before the signing of the CPA, USAID had supported peacebuilding efforts in the South through the Sudan Peace Fund (SPF) and the South Sudan Transition Initiatives (SSTI) program. SI fielded a five-member evaluation team to conduct the final evaluation for both programs to determine the effectiveness of the SSTI and SPF programs in supporting peace in Southern Sudan with an underlying interest in determining how these programs have supported USAID's overall Fragile States Strategy. The teams were sent to South Sudan to cover six geographical regions: Bahr-El-Ghazel, Eastern Equatoria, Nuba Mountains, Upper Nile, Blue Nile and Juba. Through document review, key interviews in Washington, Nairobi and Sudan with key program stakeholders and communities, the team produced a final evaluation report highlighting findings and recommendations which assisted in documenting lessons learned and best practices for future peacebuilding projects and work in fragile states, and aided OTI in decisions regarding future collaboration with implementing partners.
-
Democratic Republic of Congo and Burundi—Joint Final Evaluation: USAID/OTI 2005-2006
In early 2004, USAID's Office of Transition Initiatives (OTI) launched programs in Burundi and the Democratic Republic of Congo (DRC). The programs addressed the post-conflict challenge of integrating ex-combatants, internally displaced peoples and victims of rape into civilian life. In 2005, as OTI prepared to handover these program activities to the respective USAID Missions and other donors, SI provided OTI and USAID with an assessment of the relevance, effectiveness and lessons learned from OTI's programming in DRC and Burundi in order to strengthen the OTI process. The evaluation methodology combined archival examination and in-depth interviews of key participants with field visits to project activity sites. OTI is currently using lessons-learned from the evaluation to improve the overall impact of future program design and implementation.
-
South Sudan Peace-building Program—Final Evaluation: USAID/OTI 2005-2006
In late 2002 and early 2003, USAID's Office of Transition Initiatives (OTI) designed the South Sudan Transition Initiative (SSTI) program in close collaboration with the USAID Democracy and Governance office and the Sudan Peace Fund (SPF) program. Since then, both SPF and SSTI have provided conflict resolution expertise in areas vulnerable to violence. Their combined effort aimed to create the conditions necessary for rehabilitation and to broaden space for governance. Given the synergy between the SPF and SSTI programs, Social Impact carried out a joint program review in order to report findings, conclusions and lessons learned from USAID/OTI's peace building programs in South Sudan. The evaluation strategy was a mixed methodology that includes a combination of qualitative and quantitative data collection, interviews with key stake-holders and extensive field visits to grantees.
-
Afghanistan—Final Evaluation of Recovery and Rehabilitation Program: USAID/OTI, 2005
The goal of the OTI/Afghanistan program is to support the process of recovery, rehabilitation and political development. The purpose of the final evaluation was to provide OTI and USAID with an assessment of the relevance, effectiveness and lessons learned from OTI's program in Afghanistan. In so doing, this evaluation explored the extent to which OTI's efforts contributed to Afghanistan's transition, the extent to which it promoted government legitimacy and whether OTI's use of participatory democratic processes increase citizen's connections to each other and to local authorities. The evaluation methodology combined a desk review and in-depth interviews of key participants with field visits to project sites. OTI is currently using lessons-learned from the evaluation to improve the impact of future programs.
-
Israel, West Bank and Gaza-Impact Evaluation of Peace-building Program: Seeds of Peace, 2005
The purpose of the Seeds of Peace (SOP) program is to equip Israeli and Palestinian youth to build a future of peaceful coexistence. SI's evaluation assessed the impact of the program and developed recommendations for program improvement. SI made use of its new M&E Toolkit for Fragile States to plan and conduct a mixed method evaluation using a quasi-experimental design. Evaluation findings-evidencing deep attitudinal and behavior changes among program participants-are being used to by SOP to sharpen program strategy, improve post-program support to Seeds and to demonstrate results.
-
Kazakhstan, Kyrgyzstan and Turkmenistan-Appreciative Mid-term Evaluation of the Civil Society Support Initiative (CSSI): Counterpart International, 2005
The CSSI aims to strengthen civil society networks in Central Asia as part of a process of democratic reform. The purpose of the mid-term evaluation was to measure and improve progress toward the development of Civil Society Support Centers (CSSCs) and to build Counterpart's capacity to undertake similar assessments. SI used a highly participatory mixed-methods approach combining Appreciative Inquiry-an asset-based evaluation method-key informant interviews and customer satisfaction surveys. The evaluation created common strategies for improving services to client organizations and a strong commitment to action along these lines on part of the CSSCs.
-
New Monitoring and Evaluating Toolkit for Fragile States Programs: USAID/OTI, 2005
Working in failed and fragile states OTI needed a specialized toolkit that would help it to better monitor and evaluate its programs and to do so quickly. In response, SI developed a Monitoring and Evaluation (M&E) toolkit for OTI-the first toolkit of its kind, specifically created for work in fragile states. The Toolkit provides: i) practical guidance for designing performance monitoring systems, ii) illustrative objectives and indicators for programs in fragile states; and iii) step-by-step guidance on use of 25 innovative M&E tools proven effective in fragile states.
-
Angola-Final Evaluation of USAID/OTI's Angola Program, 2004
OTI's program in Angola aimed to help the country make a successful transition to a peaceful and democratic society. SI's final evaluation of the Angola program used a mix of both qualitative and quantitative methods to assess the extent to which OTI met its stated goal and objectives, the overall management and operation of the program, lessons learned, and ways OTI could improve future programs. Key recommendations centered on the need for OTI do more to build the capacity of NGOs/CSOs for advocacy and engagement, expand responsibilities for local staff and to ensure stronger communication and coordination between OTI and its implementing partner.
-
Global Evaluation of World Bank Development Marketplace, 2004
The Development Marketplace (DM) supports innovative grassroots development ideas which can then be expanded or replicated. SI evaluated the DM's global 2002 grants program and assessed 30 projects from 26 countries spread across 10 sectors. Using local and international enumerators, projects were rated according to eight thematic areas: innovation; objectives; knowledge generation and dissemination; partnerships; capacity building and technical assistance; sustainability; results spread and replication; and social entrepreneurship. As a result of the evaluation DM has extended the duration of its grants and is making other improvements in building program effectiveness through better capacity-building support for grantees.
-
Nigeria—Assessment of USAID Rural Sector Enhancement Program (RUSEP), 2004
The primary objective of RUSEP is to strengthen agricultural enterprises in targeted areas of Nigeria. SI evaluated RUSEP for the purposes of documenting lessons learned, assessing program impact at the household and community levels, quantifying impact on agricultural productivity and access to credit, and identifying program components with the greatest potential for sustainability and replication. USAID/Nigeria used the mixed-method assessment to revise its five-year strategy with a stronger focus on rural livelihoods, farmers in remote areas and support for agricultural extension services.
-
Afghanistan-Mid-Term Assessment of USAID/OTI's Afghanistan Program, 2003
The aim of OTI's program in Afghanistan is to support the Afghan government in the process of rehabilitation and political stabilization. SI conducted a Mid-term Assessment in collaboration with a female Afghan social scientist who had reliable access to local communities to discuss project strengths and weaknesses. Using SI's findings and recommendations, OTI worked on sharpening its support for increasing the capacity of the Afghan government, state and independent media, as well as building confidence of local citizens in democratic processes through public information.
-
Macedonia-Final Evaluation of USAID/OTI Confidence Building Initiative, 2003
The aim of OTI's Confidence Building Initiative (CBI) was to mitigate conflict-targeted communities affected by ethnic violence. SI assessed the relevance, efficiency, effectiveness, impact and probable sustainability of CBI activities. Using focus groups and mini-surveys, SI was able to verify-for the first time-measurable changes such as more collaborative attitudes and an increased trust and willingness to work together in OTI supported communities. Evaluation recommendations were internalized to improve OTI's program management approach and coordination with local partners.
-
Somalia—Mid-term Evaluation of UNDP Capacity Building for Governance (CBG) Project, 2003
The goal of the CBG Project is to strengthen Somalia's public administration and government cooperation with civil society organizations. The SI evaluation assessed project progress; crosscutting issues of gender, poverty and environment; capacity building for targeted government agencies; internal project management practices; linkages with other donor financed projects; and the strategic alignment of CBG with UNDP objectives. The UNDP heavily relied upon the evaluation to redesign the CBG program and focus on its only well-performing component—strengthening cooperation with civil society organizations.
-
Participatory Evaluation
Download PDF
Improving program performance while building capacity of local institutions (Impact Note No. 7) -
Performance Improvement Programming
Download PDF
A participatory approach to strengthening program implementation (Impact Note No. 8)