Services: Program Evaluation
Evaluation is a platform for learning, accountability and focused strategy.
Program evaluation is central to organizational learning, performance improvement and accountability. Through evaluation we can determine the efficiency, relevance, effectiveness, sustainability and institutional impacts of development activities. Impact evaluation helps us to better understand the extent to which activities reach the poor and the magnitude of their effects on people's welfare.
SI provides a full range of evaluation services. Our approach is built on international standards of excellence requiring evaluations to be useful, practical, ethical and accurate. We also build the capacity of organizations to monitor and evaluate their own programs in order to better achieve their missions.
Range of Services
- Impact Evaluation-systematic assessment of effects on individual households, institutions, and the environment caused by a given development project, program or policy.
- Evaluations for Program Improvement-evaluations focused on improving projects and programs during the project life.
- Participatory and Empowerment Evaluation-evaluations with central objectives to foster participation, learning and empowerment among local stakeholder groups.
- Evaluation Capacity Development-building organizational skills, incentives and systems to use evaluation in order to achieve results.
- PROJECT details
-
Strengthening Results-Based M&E Systems for the Mindanao Trust Fund Reconstruction and Development Program (MTF-RDP), World Bank, 2008-09
With ever-increasing prospects for a peace treaty between the Moro Islamic Liberation Front and the Philippines Government, the World Bank and other donors have created the Mindanao Trust Fund for Development and Reconstruction Program (MTF-RDP). This program focuses on providing capacity building and small-scale reconstruction and development projects using a community-driven development (CDD) approach to areas in Mindanao that suffer from lack of infrastructure and development planning, displaced populations, and mistrust at the local and institutional levels. The World Bank contracted SI to review and improve MTF-RDP’s current M&E system, so that it produces timely information that can be better utilized for management decision-making, program improvement and accountability purposes. As part of this work, SI is developing and implementing a number of systematic data collection tools, including short questionnaires, interviews, mini case studies, focus groups, and a governance index. SI conducted a short interim evaluation of staff and community members in the fall of 2008. In Spring 2009, SI will conduct an extensive evaluation of the program to date. In addition, SI provides guidance and recommendations on efforts to harmonize M&E among major donors working in Mindanao. To accomplish all of these objectives, SI employs one local full-time M&E Advisor in Mindanao, as well as three M&E experts on a short-term basis.
-
Monitoring and Evaluation Technical Support to Government of Rwanda in Implementing the ‘Vision 2020 Umurenge Programme’: DFID, 2008-2010
Social Impact is currently supporting the Government of Rwanda (GoR) to enhance the monitoring and evaluation capacity of the team implementing ‘Vision 2020 Umurenge Programme’ (VUP 2020), the Rwandan government’s program to eradicate extreme poverty. The SI Team, composed of an international and local consultant, is providing technical expertise to support the National Team Leader/Coordinator, the national M&E specialist and the district and sector officials to ensure effective monitoring and evaluation of the VUP 2020 programme. Specifically, they will be developing the methodology for a monitoring and impact evaluation system; completing a baseline and baseline report; establishing a monitoring system and quarterly reporting system; completing an impact evaluation and report; conducting an analysis of interventions and lessons learned; and most importantly, building the capacity of the VUP team to carry out M&E activities.
-
Evaluation of Border Security Programs in Kyrgyz Republic and Tajikistan, U.S. Department of State, 2008-2009
US assistance goals in Tajikistan and Kyrgyzstan are aimed at promoting democratic processes, including the ability of these countries to secure their own borders against human trafficking, drug smuggling, weapons of mass destruction, and terrorist activities.
When the U.S. State Department’s Bureau of European and Eurasian Affairs required an assessment of current programs in order to increase the effectiveness of U.S. support for border security in Central Asia, it called on experts from Social Impact. By achieving a better understanding of the effectiveness of USG-funded border security activities in the Kyrgyz Republic and Tajikistan, we were able to identify lessons learned that would benefit donor programs not only in the two countries studied, but in the region as a whole. The two-person team we deployed reflected our experience and insights in Central Asia with U.S. programming, host governments, local workers, and other international donors. Using such methods as document reviews, structured interviews, focus groups and site visits, our specialists provided relevant insights for on-the-ground managers to improve implementation of existing programs and offer policy-relevant learning for the improvement of future programming in this essential sector.
-
El Salvador-Impact Evaluation Design & Implementation Services for Infrastructure Components of the Millennium Challenge Account (MCA): Millennium Challenge Corporation (MCC), 2007-2012
As the only small business providing impact evaluation services to the Millennium Challenge Corporation (MCC), Social Impact was awarded Task Order #01: Impact Design and Implementation Services- El Salvador Infrastructure, a five-year contract. The activities we are evaluating in the Northern Region of El Salvador include:
- Roads Connectivity – a Transnational Highway and a Network of Connecting Roads;
- Rural Electrification – an expansion estimated to reach an additional 235,000 individuals and 47,000 households;
- Water and Sanitation – a component to improve water systems for 90.000 individuals and provide sanitation services for 50,000 inhabitants; and
- Community Infrastructure – construction and rehabilitation of small roads, drainage works, retaining walls, pedestrian crossing, small bridges and other community infrastructure.
During the first year of the program, the SI Team (SI, IFPRI and RTI) worked extensively with MCC, MCA-El Salvador and other key stakeholders to develop and finalize evaluation methodologies and an implementation plan. This has included providing presentations both on impact evaluations and our proposed methodologies. For the Connectivity and Electrification project, SI also has designed a baseline survey, determined the sample size, and trained enumerators to carry it out. In November 2008, the survey was implemented and is currently being given to roughly 5,400 households. A follow-up survey will be given in November 2010 and a final in November 2011. Frequent data quality checks are on-going. Later this year, SI will analyze and write a report on the baseline data. We will also design the voucher system for Rural Electrification. For Water and Sanitation, a draft baseline survey with a proposed sampling of roughly 3,400 household has been produced. SI is currently finalizing the survey and selecting the households. In Spring 2009, SI will train enumerators and implement the survey. It is expected that this survey will also be given in April 2011 and April 2012. SI will also design a Community Infrastructure survey. In addition, in the coming years SI will provide quarterly reports, oversee data collection, analyze data, make adjustments to the methodology as necessary, conduct the analysis, and write interim and final evaluation reports.
-
Impact Evaluation Services for the Millennium Challenge Corporation, 2007-2011
SI is a lead organization on an Indefinite Delivery, Indefinite Quantity (IDIQ) contract to provide strong impact evaluation services to the Millennium Challenge Corporation (MCC) for their economic growth and poverty reduction programs in developing countries. With subcontractors Research Triangle Institute, Inc (RTI) and International Food Policy Research Institute (IFPRI), our impact evaluations will enable MCC to demonstrate the economic and social return on their investments and provide MCC, host country policy makers and donors with strong evidence for making decisions about how to allocate development resources. SI provides MCC with evaluation approaches using randomized control trials, and in instances where these are not feasible, SI provides the most rigorous quasi-experimental designs as alternatives. SI works closely with and provides capacity building training for the MCC and local counterparts during each phase of the project to ensure the highest quality work and a feeling of shared ownership over the evaluation process and products. SI also promotes the utilization of MCC impact evaluations by national counterparts and develops a learning culture to disseminate and utilize the findings and lessons more broadly. SI was also awarded the first multi-year task order to provide impact evaluation services for MCC's new program in El Salvador.
-
Armenia Media Sector Mid-term Evaluation, USAID/IREX, 2007
The primary mission of the Core Media Support Program of Armenia (CMSPA), a four-year program implemented by IREX and funded by USAID, is to foster self sustainability in the Armenian media sector. The main element of the program is designed to assist commercially viable targeted independent media outlets through the provision of loans. As IREX reached the mid-point of the program, it commissioned SI to conduct a mid-term evaluation of the program to gain an independent assessment of the impact on targeted Armenia media outlets (and the wider media sector) of prior and on-going program initiatives and recommendations for possible future initiatives. The SI team was asked to carefully assess the program and develop a set of practical recommendations for program improvement aimed at maximizing program results during the remaining time of the CMSPA program. SI presented its findings and recommendations through debriefs for IREX field and HQ staff, as well as via a final evaluation report. The team conducted its research through a desk review, content analysis and a field trip to Armenia. IREX is currently implementing these recommendations to improve program performance, especially in the areas of data analysis, reporting and extending the reach of the loan program to additional regions in Armenia.
-
Final Evaluation of Emergency Capacity Building Project, 2007
The Emergency Capacity Building Project (ECB), funded by the Bill and Melinda Gates Foundation, is a cooperative undertaking of seven large humanitarian agencies (CARE, Save, CRS, Oxfam-UK, IRC, Mercy Corps and NETHOPE) aimed at improving their emergency response capacity and in particular the speed, quality and effectiveness in saving lives, improving welfare and protecting the rights of people in emergencies. SI conducted the final evaluation for the project focusing on a review of project design, implementation, effectiveness, results and outcomes while reflecting on processes used to promote partnerships and organizational learning within the humanitarian sector. The evaluation was also planned to inform future design and operations of emergency capacity building projects. Using a mixed method evaluation approach, including extensive interviewing, desk review and an innovative partnership survey, SI supported the ECB team to consolidate an understanding of results and lessons learned from their first phase of work together and to feed this learning directly into the design of Phase II of the project.
-
Colombia - Strengthening SINERGIA, the M&E System of the Colombian Government, 2007-08
SI worked with the Colombian government to improve the rigor, sustainability, and capacity of their government-wide M&E system SINERGIA, recognized as one of the leading M&E systems in Latin America. Specifically, the SI Team reviewed and rated the evaluation methodologies currently employed; offered additional techniques to strengthen their evaluation work; developed a system for prioritizing evaluations based on the magnitude of expected effects, the number of people affected and the budget; reviewed the adequacy of the current incentive system; and provided recommendations on how to promote greater utilization of M&E information by all government entities. As the Colombian government was very satisfied with our report, we were invited to and ultimately did present our findings at a conference in Bogota co-sponsored by the World Bank.
-
Impact Evaluation Services for the Millennium Challenge Corporation, 2007-2011
SI is a lead organization on an Indefinite Delivery, Indefinite Quantity (IDIQ) contract to provide strong impact evaluation services to the Millennium Challenge Corporation (MCC) for their economic growth and poverty reduction programs in developing countries. With subcontractors Research Triangle Institute, Inc (RTI) and International Food Policy Research Institute (IFPRI), our impact evaluations will enable MCC to demonstrate the economic and social return on their investments and provide MCC, host country policy makers and donors with strong evidence for making decisions about how to allocate development resources. SI provides MCC with evaluation approaches using randomized control trials, and in instances where these are not feasible, SI provides the most rigorous quasi-experimental designs as alternatives. SI works closely with and provides capacity building training for the MCC and local counterparts during each phase of the project to ensure the highest quality work and a feeling of shared ownership over the evaluation process and products. SI also promotes the utilization of MCC impact evaluations by national counterparts and develops a learning culture to disseminate and utilize the findings and lessons more broadly. SI was also awarded the first multi-year task order to provide impact evaluation services for MCC's new program in El Salvador.
-
Sri Lanka—Final Evaluation of the Sri Lanka Transition Initiative: USAID/OTI, 2007
The Sri Lanka Transition Initiative (SLTI) was launched in February 2003 to support peace talks between the Government of Sri Lanka and the Liberation Tigers of Tamil Eelam (LTTE) through generating greater support for a negotiated peace settlement to end the longstanding internal conflict. The objectives of this program were to: i) change attitudes sustaining the conflict; and ii) mobilize and link peace constituencies through local-level activities promoting collaborative decision-making. Social Impact conducted a final evaluation of the program that documented accomplishments and lessons learned for USAID. As a result of the document review, extensive interviews and a nearly three-week field mission to Sri Lanka, the evaluation team analyzed the impact of the SLTI program as well as lessons learned and best practices to benefit future programming. The evaluation was used by OTI to develop a strategy for the final phase of the program. The report highlighted that the SLTI substantially contributed to livelihoods, local infrastructure and, in some cases, the quality of life in Sri Lanka by responding to the challenge of the tsunami and post-tsunami rehabilitation period with commitment and ingenuity in the hardest hit areas.
-
Assistance to Russian Orphans 2 Program Evaluation: USAID, 2006
In response to the challenges of Russia's ill-equipped and ineffective child welfare system, the Assistance to Russian Orphans 2 (ARO2) program, sponsored by the USAID and implemented by IREX and its local partner, the National Foundation for the Prevention of Cruelty to Children (NFPCC), aims to increase the use of improved health and child welfare practices through the implementation of new child abandonment prevention and family-based care services. SI conducted the ARO2 Evaluation in the Russian Far East, Siberia and Moscow to develop practical recommendations for program improvement aimed at consolidating program results and to improve program monitoring and evaluation systems so that results can be better documented during the next phase of the program, ARO3. The report thus evaluated the strengths and challenges of ARO2 to institutionalize and consolidate the gains that were made during ARO2 and to ensure that mechanisms are in place for their spread throughout the Russian Federation during ARO3.
-
Liberia—Final Evaluation of USAID/OTI Program, 2006
The Liberia Transition Initiative was launched in February 2004 to help advance an inclusive, peaceful transition in Liberia following 14 years of civil war that ended with the 2003 Comprehensive Peace Agreement. Through this program, OTI worked to restore critical transitional governance and civil society functions; increase the understanding of key political transition issues; and promote community integration and peaceful resolution of conflict. SI performed the final evaluation of this program which included activities that focused on youth and community infrastructure development. Specifically, the evaluation was designed to explore the extent to which OTI's efforts advanced the prospects for an inclusive and peaceful political transition in Liberia. By adopting a context-specific approach that takes into account the needs, interests and indicators for success that are meaningful to the beneficiaries of the program, SI produced an evaluation report that was used by OTI to develop the second phase of the Liberia Transition Initiative. The report highlighted OTI's vital ability to bring immediate evidence of the benefits of peace to a war stressed population, and to reinforce that peace by using grants to foster effective governance and social and political cooperation.
-
Strategy and Impact of the Iraq Transition Initiative: USAID/OTI, 2006
The Iraq Transition Initiative (ITI) was launched in April 2003 to work closely with the Iraqi Interim Government and the United States Embassy in Iraq. The objectives of ITI were to identify and fill crucial gaps in the US Government's assistance efforts at the national and local levels and increase public support for the Interim Government. Activities centered on the disbursement of small grants to local Iraqi groups and institutions. Since the program's beginning, ITI issued approximately 5,000 small grants totaling $335 million. Social Impact conducted a final evaluation of the impact of this project. As a result of the document review and extensive interviews, SI analyzed quantitative and qualitative information that identified the strengths and weakness of the program as well as lessons learned and best practices to benefit future programming. The findings of the evaluation are being used by OTI to demonstrate results of the program and lessons learned are being used to improve OTI-military relations, better align strategic and programmatic objectives and to improve the results-orientation and monitoring of future OTI projects in transition countries.
-
Sudan—Final Evaluation: The Sudan Peace Fund and the South Sudan Transition Initiative Program: USAID/Office of Transition Initiatives (OTI), 2006
The 2005 Comprehensive Peace Agreement (CPA) signed between the Government of Sudan and the Sudanese People's Liberation Movement/Army presented an historic opportunity to end 50 years of fighting between the North and the South. Peace, however, was contingent upon internal Southern stability, which had been undermined by 23 years of inter-ethnic fighting. For three years before the signing of the CPA, USAID had supported peacebuilding efforts in the South through the Sudan Peace Fund (SPF) and the South Sudan Transition Initiatives (SSTI) program. SI fielded a five-member evaluation team to conduct the final evaluation for both programs to determine the effectiveness of the SSTI and SPF programs in supporting peace in Southern Sudan with an underlying interest in determining how these programs have supported USAID's overall Fragile States Strategy. The teams were sent to South Sudan to cover six geographical regions: Bahr-El-Ghazel, Eastern Equatoria, Nuba Mountains, Upper Nile, Blue Nile and Juba. Through document review, key interviews in Washington, Nairobi and Sudan with key program stakeholders and communities, the team produced a final evaluation report highlighting findings and recommendations which assisted in documenting lessons learned and best practices for future peacebuilding projects and work in fragile states, and aided OTI in decisions regarding future collaboration with implementing partners.
-
Democratic Republic of Congo and Burundi—Joint Final Evaluation: USAID/OTI 2005-2006
In early 2004, USAID's Office of Transition Initiatives (OTI) launched programs in Burundi and the Democratic Republic of Congo (DRC). The programs addressed the post-conflict challenge of integrating ex-combatants, internally displaced peoples and victims of rape into civilian life. In 2005, as OTI prepared to handover these program activities to the respective USAID Missions and other donors, SI provided OTI and USAID with an assessment of the relevance, effectiveness and lessons learned from OTI's programming in DRC and Burundi in order to strengthen the OTI process. The evaluation methodology combined archival examination and in-depth interviews of key participants with field visits to project activity sites. OTI is currently using lessons-learned from the evaluation to improve the overall impact of future program design and implementation.
-
South Sudan Peace-building Program—Final Evaluation: USAID/OTI 2005-2006
In late 2002 and early 2003, USAID's Office of Transition Initiatives (OTI) designed the South Sudan Transition Initiative (SSTI) program in close collaboration with the USAID Democracy and Governance office and the Sudan Peace Fund (SPF) program. Since then, both SPF and SSTI have provided conflict resolution expertise in areas vulnerable to violence. Their combined effort aimed to create the conditions necessary for rehabilitation and to broaden space for governance. Given the synergy between the SPF and SSTI programs, Social Impact carried out a joint program review in order to report findings, conclusions and lessons learned from USAID/OTI's peace building programs in South Sudan. The evaluation strategy was a mixed methodology that includes a combination of qualitative and quantitative data collection, interviews with key stake-holders and extensive field visits to grantees.
-
Afghanistan—Final Evaluation of Recovery and Rehabilitation Program: USAID/OTI, 2005
The goal of the OTI/Afghanistan program is to support the process of recovery, rehabilitation and political development. The purpose of the final evaluation was to provide OTI and USAID with an assessment of the relevance, effectiveness and lessons learned from OTI's program in Afghanistan. In so doing, this evaluation explored the extent to which OTI's efforts contributed to Afghanistan's transition, the extent to which it promoted government legitimacy and whether OTI's use of participatory democratic processes increase citizen's connections to each other and to local authorities. The evaluation methodology combined a desk review and in-depth interviews of key participants with field visits to project sites. OTI is currently using lessons-learned from the evaluation to improve the impact of future programs.
-
Israel, West Bank and Gaza-Impact Evaluation of Peace-building Program: Seeds of Peace, 2005
The purpose of the Seeds of Peace (SOP) program is to equip Israeli and Palestinian youth to build a future of peaceful coexistence. SI's evaluation assessed the impact of the program and developed recommendations for program improvement. SI made use of its new M&E Toolkit for Fragile States to plan and conduct a mixed method evaluation using a quasi-experimental design. Evaluation findings-evidencing deep attitudinal and behavior changes among program participants-are being used to by SOP to sharpen program strategy, improve post-program support to Seeds and to demonstrate results.
-
Kazakhstan, Kyrgyzstan and Turkmenistan-Appreciative Mid-term Evaluation of the Civil Society Support Initiative (CSSI): Counterpart International, 2005
The CSSI aims to strengthen civil society networks in Central Asia as part of a process of democratic reform. The purpose of the mid-term evaluation was to measure and improve progress toward the development of Civil Society Support Centers (CSSCs) and to build Counterpart's capacity to undertake similar assessments. SI used a highly participatory mixed-methods approach combining Appreciative Inquiry-an asset-based evaluation method-key informant interviews and customer satisfaction surveys. The evaluation created common strategies for improving services to client organizations and a strong commitment to action along these lines on part of the CSSCs.
-
New Monitoring and Evaluating Toolkit for Fragile States Programs: USAID/OTI, 2005
Working in failed and fragile states OTI needed a specialized toolkit that would help it to better monitor and evaluate its programs and to do so quickly. In response, SI developed a Monitoring and Evaluation (M&E) toolkit for OTI-the first toolkit of its kind, specifically created for work in fragile states. The Toolkit provides: i) practical guidance for designing performance monitoring systems, ii) illustrative objectives and indicators for programs in fragile states; and iii) step-by-step guidance on use of 25 innovative M&E tools proven effective in fragile states.
-
Angola-Final Evaluation of USAID/OTI's Angola Program, 2004
OTI's program in Angola aimed to help the country make a successful transition to a peaceful and democratic society. SI's final evaluation of the Angola program used a mix of both qualitative and quantitative methods to assess the extent to which OTI met its stated goal and objectives, the overall management and operation of the program, lessons learned, and ways OTI could improve future programs. Key recommendations centered on the need for OTI do more to build the capacity of NGOs/CSOs for advocacy and engagement, expand responsibilities for local staff and to ensure stronger communication and coordination between OTI and its implementing partner.
-
Global Evaluation of World Bank Development Marketplace, 2004
The Development Marketplace (DM) supports innovative grassroots development ideas which can then be expanded or replicated. SI evaluated the DM's global 2002 grants program and assessed 30 projects from 26 countries spread across 10 sectors. Using local and international enumerators, projects were rated according to eight thematic areas: innovation; objectives; knowledge generation and dissemination; partnerships; capacity building and technical assistance; sustainability; results spread and replication; and social entrepreneurship. As a result of the evaluation DM has extended the duration of its grants and is making other improvements in building program effectiveness through better capacity-building support for grantees.
-
Nigeria—Assessment of USAID Rural Sector Enhancement Program (RUSEP), 2004
The primary objective of RUSEP is to strengthen agricultural enterprises in targeted areas of Nigeria. SI evaluated RUSEP for the purposes of documenting lessons learned, assessing program impact at the household and community levels, quantifying impact on agricultural productivity and access to credit, and identifying program components with the greatest potential for sustainability and replication. USAID/Nigeria used the mixed-method assessment to revise its five-year strategy with a stronger focus on rural livelihoods, farmers in remote areas and support for agricultural extension services.
-
Afghanistan-Mid-Term Assessment of USAID/OTI's Afghanistan Program, 2003
The aim of OTI's program in Afghanistan is to support the Afghan government in the process of rehabilitation and political stabilization. SI conducted a Mid-term Assessment in collaboration with a female Afghan social scientist who had reliable access to local communities to discuss project strengths and weaknesses. Using SI's findings and recommendations, OTI worked on sharpening its support for increasing the capacity of the Afghan government, state and independent media, as well as building confidence of local citizens in democratic processes through public information.
-
Macedonia-Final Evaluation of USAID/OTI Confidence Building Initiative, 2003
The aim of OTI's Confidence Building Initiative (CBI) was to mitigate conflict-targeted communities affected by ethnic violence. SI assessed the relevance, efficiency, effectiveness, impact and probable sustainability of CBI activities. Using focus groups and mini-surveys, SI was able to verify-for the first time-measurable changes such as more collaborative attitudes and an increased trust and willingness to work together in OTI supported communities. Evaluation recommendations were internalized to improve OTI's program management approach and coordination with local partners.
-
Somalia—Mid-term Evaluation of UNDP Capacity Building for Governance (CBG) Project, 2003
The goal of the CBG Project is to strengthen Somalia's public administration and government cooperation with civil society organizations. The SI evaluation assessed project progress; crosscutting issues of gender, poverty and environment; capacity building for targeted government agencies; internal project management practices; linkages with other donor financed projects; and the strategic alignment of CBG with UNDP objectives. The UNDP heavily relied upon the evaluation to redesign the CBG program and focus on its only well-performing component—strengthening cooperation with civil society organizations.
-
Participatory Evaluation
Download PDF
Improving program performance while building capacity of local institutions (Impact Note No. 7) -
Performance Improvement Programming
Download PDF
A participatory approach to strengthening program implementation (Impact Note No. 8)