External Evaluation of Armenia Round Table Project “Sustainable and participatory community development in Armenia, continuation”

TERMS OF REFERENCE

for External Evaluation of Armenia Round Table Project “Sustainable and participatory community development in Armenia, continuation” (E-ARM-2022-0303)

World Council of Churches Armenia Inter-Church Charitable Round Table Foundation (hereinafter ART) is seeking evaluators to undertake external evaluation of ART “Sustainable and participatory community 5development in Armenia, continuation” project 2023–2025. 

1. Introduction

ART is an ecumenical organization established by the Armenian Apostolic Church and World Council of Churches in 1996. ART brings together Armenian churches, local non-governmental organisations, church-related partners and international ecumenical agencies with a shared commitment of working together to address some of the fundamental social and economic problems of the country. ART office is a service-providing and facilitating body between local churches and NGOs, national and international agencies. ART is active in the fields of integrated community development and diakonia, capacity building of civil society, emergency preparedness and response. 
Following the last project evaluation taken place in 2022, ART developed the present project for 01 January 2023 — 31 December 2025. However, due to sharp depreciation of EUR against Armenian dram, the project developed a considerable loss and in consultation and with approval of Brot für die Welt (Bread for the World, BfdW, Germany), it was decided to shorten project period and end it on June 30, 2025. 
The project is financially supported by different organizations, such as BfdW, Armenian International Women’s Association (AIWA, USA), FAP, Switzerland, International Orthodox Christian Charities (IOCC, Switzerland), UNFPA-Armenia, private donations from Armenian diaspora, etc., and by cash contributions of local implementing partners. ART prioritizes participatory approach and consensus building in its work and, therefore, encourages partners’ involvement in project design, implementation, monitoring and evaluation. Consequently, stakeholders relevant to this evaluation are the Church and Church related community centres, local communities, partner NGOs, local authorities, and funding partners.
The development goal of the project “Sustainable and participatory community development in Armenia, continuation” (E-ARM-2022-0303) is to contribute to improvement of livelihoods and civic resilience of people living in Armenian regions.  

Primary beneficiaries of the project are vulnerable populations, unemployed, women-headed households, unemployed young people, children from large, one-parent and socially disadvantaged families, single elderly, refugees with scarce economic opportunities, and people with disabilities from the regions  Aragatsotn, Ararat, Armavir, Gegharkunik, Kotayk, Lori, Shirak, Syunik and Tavush.

Project objectives are:
Objective 1: Disadvantaged youth, women and men in the targeted communities have improved livelihood.
Objective 2: Targeted communities engage in local decision-making.

To achieve Objective 1, the project activities are:
Activity: Activity 1.1 Develop organizational capacity of ART supported centers
Activity 1.2 Support livelihoods and development opportunities of vulnerable populations 
 
To achieve Objective 2, the project implements the following activities:
Activity 2.1 Civic education 
Activity 2.2 Civic activism

Project Outputs are:
Output 1.1: Capacity of 3 centers strengthened for provision of supplementary education
Output 1.2: 650 young people and adults (women) improved their marketable skills, employability and economic activity 
Output 1.3: 1,110 school age children and their families improved personal development  
Output 2.1: 90 targeted youth has increased transversal skills and are informed about civic activism, human rights, equality and citizenship
Output 2.2: 18 micro grants provided to youth for civic engagement and awareness raising

The project operates at the background of a challenging and specific context that Armenia represents. ART worked in realities of aftermaths of a 44-day war over Nagorno-Karabakh in September 2020, military attack on Armenia in September 2022 and arrival of almost entire population of Nagorno Karabakh more than 101, 000 people in September 2023 after military attack by Azerbaijan in the fear of ethnic cleansing. The continuous emergencies and humanitarian crisis following them seriously affected all aspects of life in Armenia including work of civil society organizations. ART and its partners, social centres have been responsive to the emergencies and continued their work maintaining along with routine social and educational activities, delivery of emergency/relief aid and early recovery support to war affected and refugees.

2. Scope, cause and objective of the evaluation

To learn from its experiences how relevant, effective, efficient, sustainable, coherent and impactful remains its work and to maintain accountability and transparency of its work, ART plans to undertake the external evaluation of its three-year project (2023 – 2025/6). The findings and recommendations from this external evaluation shall provide important information for development of the next project 2025/7–2028/6 for ART and Brot für die Welt. 

The evaluation of the “Sustainable and participatory community development in Armenia, continuation” project will cover the following:
— Process: assess whether the project is being implemented as planned, to what extent the intended target population is being reached, and identify major challenges and successful practices, strategies and/or amendments associated with the project implementation;
— Outcome: determine the effectiveness and efficiency of the expected changes in the lives of beneficiaries occurred and whether these changes can be attributed to the project;
— Assess possible indirect effects of the project;
— Offer recommendations and advise on how to consolidate the results of the project until its finalisation and  increase effectiveness and influence of a potential future project and its sustainability;
— Examine, whether additional measures/activities need to be carried out within the current project with existing target groups to ensure sustainability and flexibility in view of changing contexts;
— To assess the sustainability of project outcomes.

The scope of this project evaluation covers the period from January 2023 to June 2025, it shall examine and evaluate achievements of the project for 2023 and 2024, with consideration that time interval is needed for some results to take place, e.g. with income generation, as the project period is shortened and evaluation will take place at the end of the project second year. When making the analysis and providing recommendations, the evaluators should take into account this aspect, as well as dramatic social, political, economic changes in the Armenian context and regional realities.

3. Key questions

The key evaluation criteria against which accomplishments of the project will be measured are defined by six OECD DAC criteria (Relevance, Coherence, Effectiveness, Efficiency, Sustainability, Impact). With reference to the DAC criteria, the following questions are to be answered:

Relevance: 
— To which extent do ART’s project objectives and measures meet the needs of its beneficiaries, local partners and the partners’ beneficiaries? Specifically, how relevant are the five launched pilot elements of the project, namely the introduction of paid services in the social centers, the introduction of a mobile social center, the expansion of ART into the Ararat and Syunik regions, the activation of youth as agents of change, and the start of initial funding for self-employment as well as practical internships for skills and professions taught in the social centers?
Is the project relevant in the context of recent changes in the country? To which extent did ART adapt to changes in the project context? Coherence:
— To what extent does the project fit in with other measures and projects in the country, generally and more specifically with regard to displaced and war-affected people?
— In which way does the project complement activities of local, regional and/or national government and affiliated agencies?
— What is ART’s specific role in the Armenian context at the edge of the Armenian Churches, its affiliated institutions and the Armenian civil society?
— How useful are the outputs to the needs of the direct beneficiaries and other stakeholders (Armenian Church, church related social centres, non-governmental organizations, local authorities)?
— 
Effectiveness:
— To which extent will the two objectives of the project foreseeably be achieved? 
— Which further steps must be taken in order to achieve the objectives until the end of the project?
— To which degree do the activities meet the objectives and planned results set out in the project? How well the activities align with the objectives and planned outcomes?
— How useful are the outputs to the needs of the direct beneficiaries and other stakeholders (Armenian Church, church related social centres, non-governmental organizations, local authorities)?
— How well do project components contribute to the empowerment of local capacities? Specifically, to which extent have beneficiaries of the supported social centers and micro business schems improved their employability and income situation? What role does the out-of-school education play in the life of its beneficiaries? 
— What are good practices that ART has developed? Which activities particularly functioned well in terms of bringing about outcomes and which activities did not generate sufficient results?
— How does ART fulfil its new role as an implementing organization for the micro grant component? Does it have the capacity to do so?
— To which extent has ART taken up and implemented recommendations from the external evaluation of the previous project phase from 2022?
Efficiency:
— Is the project run in an efficient way?
— How well does the project use its resources to produce outcomes and achieve impact? How economically viable are the project incurred costs in relation to the deliverables and effects produced?
— What is the project status with respect to target outputs in terms of quantity, quality and timeliness?
— How reasonable was the shortening of the project cycle against the background of a deteriorating exchange rate?
Sustainability:
— Will the results achieved (intended positive changes) have a lasting effect?
— Which positive changes and impacts are likely to continue beyond the project period?
— What potential steps have to be taken in a future project to enhance sustainability of the achieved results?
— How has the sustainability of the nine supported social centers developed over time and can be assessed at this point of time? What potential steps should be taken achieve their financial independence from ART?
— How have project activities geared at the organisational development of ART contributed to the organisation’s sustainability?
Impact:
— Does the project contribute to the attainment of the overall development goal?
— Were there any unforeseen positive or negative effects of the activities?

Cross cutting issues on gender, inclusion of displaced, refugees, war affected, and environmental impact should be examined, too, and eventually feed into conclusions, recommendations, lessons learned.

The final evaluation report should answer the questions set out above, however, not be limited to these questions only, but include also those identified at the kick off meeting and during the course of evaluation. 

4. Evaluation design/methods

The methods and evaluation design should take into account OECD DAC quality standards for Development Evaluation , and cross cutting issues on gender, inclusion and environmental impact. 

The overall evaluation approach will be the following: 
— Preparation phase: The exercise will entail secondary data review (including project proposal and progress reports, evaluation report from April 23, 2022 of the former project), a kick off meeting, writing an inception report for evaluation stating the sources of information and tools for collection (design of data collection instruments, e.g. questionnaires and interview protocols). The evaluation should be participatory and make use of different data sources, collected through multiple methods. 
— Field phase: The exercise will entail meeting key stakeholders, conducting interviews, discussions and observations.
— Analysis and debriefing phase: The findings and recommendations will be presented to ART team, ART Board of Trustees and partners at ART office. A debriefing will be organized to discuss the findings and the recommendations.
— Reporting phase: Final evaluation report will be prepared and presented to ART and Brot für die Welt for review and approval.

In order to maximize the participation of local partners and beneficiaries, interviews, group discussions and questionnaire surveys will be conducted in both Armenian and English.
To answer the evaluation questions, evaluators are expected to interview in particular relevant ART staff, local partners, and community members involved in project actions, national level policy makers, and the Church. Evaluation findings will be mainly used by ART and Brot für die Welt to improve project steering and implementation for the current as well as for upcoming project phases.
The evaluation will seek to visit the following areas: project sites, community development centers and partners (e.g. Aragatsotn, Ararat, Armavir, Gegharkunik, Lori, Shirak, Syunik and Tavush). 

5. Process of Evaluation/Time frame 

The duration of the assignment is 10-12 working days distributed over a period from November 20, 2024 to December 10, 2024. Selection announcement, concluding the Contract and sending all relevant materials regarding the desk study component of the assignment is planned in early to mid-November. Timetable and phases in the table below are approximate and may be modified.
                                                                                                                                                                                              Timetable and Phases
No Phase Activities Time
1. 

Preparation phase: Selection of consultant(s) and contracting
Review and analysis of all relevant reference documents . November 10-15 

2. Kick off and Clarification meeting November 17 
3. Submission of the Inception report for evaluation by the consultants stating the sources of information and tools for collection. November 18-20 
4. Approval of the Inception report by ART November 21-22

5.
 
Field phase: Data collection and field work November 25-29
(between 4-5 days in the field depending on weather conditions)

6. Analysis and debriefing phase: Presentation and verification of preliminary findings to ART Executive Director, staff, ART Board and partners; inputs for the planning of the follow-up project in accordance to formulated recommendations Final day of field work

7. Reporting
phase: Preparation of Draft evaluation report December 10 
8. Receipt of the Draft evaluation report by ART and Brot für die Welt
Review and request for corrective action by ART and Brot für die Welt December 10

December 18
9. Submission of the final evaluation report January 15
10. Receipt, assessment, acceptance of the Final report by ART and Brot für die Welt January 22

6. Expected products of the evaluation

The evaluators will be accountable to the commissioner and funder of the evaluation. All required reports will be written by the evaluators in English and will be provided to the ART Executive Director, ART Programme Director and Brot für die Welt.
 
The evaluators are required to submit following products:

— Inception report, to be submitted 2 days after the kick-off meeting, explaining the assignment concept, evaluation methodology, work project, timetable for the evaluation and possible restrictions. The inception report must be approved in written by ART.
— Draft of the Final evaluation report,
— A strategic planning workshop to present and verify the findings and to plan follow-up project in accordance to formulated recommendations,
— Final evaluation report, to be submitted at the end of the evaluation with a maximum extension of 20 up to 40 pages excluding annexes. The format for the presentation of the Evaluation report is given in the Annex to the ToR. Where needed it can be extended with additional chapters and sub-chapters. 

7. Composition of evaluation team, roles and key qualifications of the evaluators
The evaluation team will comprise of two external evaluators: international evaluator and national evaluator.
The evaluators will be responsible for the quality of the evaluation process and development of the final evaluation report. They will develop the evaluation methodology and tools. In addition, the evaluators are free to add to or slightly modify the above-mentioned evaluation questions, after consultation with ART and Brot für die Welt.

The national evaluator will give specific insights of the local situation. (S)he will take the lead in facilitating face-to-face meetings and interviews with all relevant stakeholders.

ART will facilitate administrative and logistical support to the evaluation team and make sure that project partners and staff are available and participate in meetings scheduled.
Required work experience, skills and competences:
Both, the international and the national evaluator must have:
— Experience in project evaluation; designing, conducting and leading evaluations, applying qualitative and quantitative evaluation methods. Experience related to institutional evaluations will be an advantage.
— Extensive knowledge of community development, capacity-building and clear understanding of church-related development and diaconal work. 
— University qualification in development, social work or other associated field of study, strong interpersonal skills, listening skills, be strong analytical thinkers, proven extensive experience in the development of project evaluations.
— Fluency in English and have excellent analytical and strong report writing skills, for the national evaluator fluency in Armenian.
— Sound knowledge of social problems in Armenia and related development work (for national evaluator).

 8. Content of evaluation offer

In response to the ToR call, qualified candidates are invited to submit applications containing following documents.
— CV including two references,
— Technical/specific offer, containing the outline of the planned evaluation process and explaining and justifying the methods to be used and specifying how relevant cross-cutting issues will be taken into consideration,
— Financial proposal, containing complete cost estimate with all fees and ancillary costs to be incurred, such as travel and local transport, accommodation, taxes and costs of meeting/s in the scope of evaluation, etc.

Applicants should submit their applications either as a team of national and international evaluators or individually by November 10 (17:00 Yerevan time) to Armenia Round Table Foundation e-mail: info@roundtable-act.am, indicating on the subject line “ART project external evaluation”.
All applicants will be notified.  

Award criteria:
Quality of the technical proposal 35%, qualification of the evaluators 30%, price 35%.
 
ANNEX
                                       Sample Structure for Evaluation Report

The points set below are the minimum components of an evaluation report; cross-cutting issues should be taken into account where practicable

Cover sheet with
— Project title
— Project number
— Implementing organisation
— Evaluator (author)
— Report date
— Region/country
— Possibly project period

Table of contents

List of abbreviations

Summary
— Short presentation of the subject matter of the evaluation, possibly including key framework conditions
— Brief information on the evaluation: Cause and objective, assessment period
— Key findings
— Key recommendations

1. Short description of the subject matter of the evaluation
— Project (components, target group, formulated objectives)
— Implementing organisation, term, donors

2. Framework conditions (only as far as relevant to the subject matter of the evaluation)
— Political, economic, ecological, societal and socio-cultural factors
— Risks to project success, assumptions/prerequisites
— Relevant activities of other organisations/private-sector companies
— Role of government actors

3. Description of the evaluation and the methodology used
— Timing of the evaluation within the course of the project
— Composition/expertise of the evaluation team
— Methodology
— Groups of people involved, number of participants
— Potential difficulties in conducting the evaluation and how to deal with them

4. Results
— 4.1 Relevance
— 4.2. Coherence
— 4.3 Effectiveness
— 4.4. Efficiency
— 4.5. Sustainability 
— 4.6. Impact

5. Recommendations for remaining project period and follow-on project (based on findings, realistic, specific and addressed)

6. General conclusions (lessons learned)
— for the project type (including exemplary nature)
— regarding the procedures and instruments

7. Appendix
— Travel and working procedure
— Sources (discussion partners, documents, specialist literature, field research etc.)
— Terms of Reference.