Стоит обратить внимание на вынесение категории «импакт» за границы оценки цепочки воздействия в описанном выше подходе GIIN. Можно говорить о том, что подобная позиция является распространенной и представляет собой тренд в современном понимании данного термина. Это, в частности, отмечает один из ведущих международных экспертов в области академических исследований оценки проектов и программ, автор концепции «4 стратегий» Алнур Эбрахим [Ebrahim, 2019].

Таким образом, если исходить из общемировых современных трендов в определении данного понятия, можно говорить о том, что термин impact становится все более «нарицательным», определяющим в целом весь спектр понятий, связанных с описанием результатов проекта, за исключением непосредственных результатов. Данное понимание мы также рекомендуем использовать в словарном корпусе для разработки модели оценки социально-экономического воздействия социальных проектов. Именно это определение использовано в данной статье.

ТЕОРИЯ ИЗМЕНЕНИЙ, КАК БАЗОВЫЙ ВОЗМОЖНЫЙ КОМПОНЕНТ МОДЕЛИ ОЦЕНКИ СОЦИАЛЬНЫХ ПРОЕКТОВ

Одним из наиболее исследованных и описанных в научной литературе базовых конструктов и оснований оценки является понятие теории изменений (ТИ) проекта, представляющей собой четкое и ясное описание предполагаемых взаимосвязей между причинами (действиями) и ожидаемыми эффектами (результатами), которых планируется достичь в рамках проекта, с указанием условий достижения каждого из уровней результатов. Теория изменений объясняет причины того, почему используются те или иные подходы к решению проблемы, а также как эти подходы приведут к желаемым изменениям.

Как отмечает А. И. Кузьмин, автор первого русскоязычного издания по теории изменений, ТИ может быть использована как инструмент решения целого спектра задач [Кузьмин А.И, Кошелева Н. А., 2014]:

• как инструмент описания результатов проекта;

• инструмент проектирования (разработки замысла проекта);

• способ визуализации замысла проекта;

• способ обоснования причинно-следственной связи между деятельностью в рамках проекта и ее результатами;

• способ определения границ воздействия проекта;

• основа для формирования системы мониторинга проекта;

• основа для оценки проекта;

• способ «архивирования» хороших практик в обобщенном виде.

Традиционно и в самом общем виде теория изменений чаще всего представляется в виде карты ожидаемых результатов, показывающей, как действия в рамках проекта приведут к достижению цели проекта, и перечня допущений или условий, при соблюдении которых сработает данная «теория». На соблюдение этих условий авторы проекта рассчитывают, тем не менее эти допущения и условия находятся за пределами их влияния.

Теория изменений – один из наиболее распространенных инструментов, используемых для описания социальной технологии проекта, для целей его последующей оценки. То есть терминологически его можно отнести именно к области оценки проектов и программ. Там же он изучался особенно активно. Так, можно встретить различные классификации видов ТИ, подходов к ее разработке, включая специализированное программное обеспечение, авторские подходы проведения оценки, основанные на ТИ, и т. п.

Разработку теории изменений можно назвать одним из универсальных инструментов, используемых при проведении оценки проектов.

ДОКАЗАТЕЛЬНЫЙ ПОДХОД В ОЦЕНКЕ СОЦИАЛЬНЫХ ПРОЕКТОВ

С темой теории изменений тесно связан другой основополагающий термин – доказательность используемой социальной технологии. Если говорить в целом, доказательный подход – это подход, основанный на применении исследовательских методов, на поиске и изучении аргументов, подтверждающих, что комплекс осуществляемых в рамках подхода действий действительно приводит к желаемым изменениям.

Доказательный подход в социальном проектировании является одним из активно развиваемых и внедряемых в настоящее время. Об этом свидетельствует рост как числа научных публикаций, так и внеакадемических дискуссий и даже создание образовательных программ в русле доказательного подхода (пример такой инициативы – магистерская программа «Доказательное проектирование и оценка программ в области управления социальными рисками в сфере детства», открытая пять лет назад в МГППУ).

В качестве обоснования важности использования доказательного подхода в социальной сфере можно привести две отмечаемые экспертами эвристики: иллюзия «чистого поля» и «абсолютного блага». Первая состоит в том, что, приступая к разработке социального проекта, предлагая новое (свое) решение социальной проблемы, авторы зачастую игнорируют существующий опыт в решении данной проблемы. Такая эвристика невозможна в ситуации, например, коммерческого проекта, где важным условием его эффективности является учет рыночного контекста. Но она вполне возможна, и вероятность ее возникновения является высокой – в общественном секторе. Изучение рынка, анализ потребностей целевой аудитории, выделение ведущих игроков и существующих технологий, анализ эффективности имеющихся решений часто не рассматриваются как обязательные при разработке социального проекта.

Теория изменений чаще всего представляется в виде карты ожидаемых результатов, показывающей, как действия в рамках проекта приведут к достижению цели проекта.

Суть второй эвристики: что «какую социальную услугу мы бы ни оказывали – это точно хорошее и правильное дело». Отсюда возникло распространенное шутливое понятие «причинять добро». Если услуга оказывается недостаточно качественно, не соответствует современным представлениям (в том числе научным) о причинах и возможных подходах к решению социальной проблемы и т. п., нельзя говорить о том, что проект действительно способствует решению данной проблемы.

Особенно остро этот вопрос ставится в ситуации, когда благополучателями проекта являются социально незащищенные группы граждан, которые по разным причинам не могут выразить свое мнение в отношении качества оказываемой им социальной услуги и в целом – отношения к проектной интервенции. Это, в частности, дети, люди с ментальной инвалидностью, бездомные и т. п. В связи с этим обоснованным является то, что тема доказательности практик в нашей стране получила свое развитие в контексте именно детских программ. В частности, группой российских экспертов создан стандарт доказательности практик в сфере детства – первый в своем роде документ, описывающий как требования, так и методические рекомендации для описания проектов в сфере детства. Рассмотрим на примере данного стандарта направления его возможного развития и использования [Стандарт доказательности практик в сфере детства (Версия № 2. 31 августа 2018)].

СПИСОК ИСПОЛЬЗОВАННЫХ ИСТОЧНИКОВ

1. Courtney Paul Conceptualising Social Value for the Third Sector and Developing Methods for Its Assessment [Журнал]. – [б.м.]: International Society for Third-Sector Research Voluntas, 2018. – https://doi.org/10.1007/s11266–017–9908–3: Т. 29.

2. Ebrahim Alnoor Measuring Social Change: Performance and Accountability in a Complex World – [Книга]. – 2019.

3. Grieco Cecilia Assessing Social Impact of Social Enterprises. Does One Size Really Fit All? [Книга]. – [б.м.]: Springer, 2015.

4. Takyi Stephen Appiah REVIEW OF SOCIAL IMPACTS ASSESSMENT (SIA): APPROACH, IMPORTANCE, CHALLENGES AND POLICY IMPLICATIONS [Журнал]. – [б.м.]: International Journal of Arts & Sciences, 2014. – Т. 07(05):217–234 (2014).

5. Ким А. И., Копыток В. К., Филиппова Ю. А., Цыганков М. В. Применение теории изменений для стратегического аудита и стратегического планирования в России / Счетная палата Российской Федерации, Центр перспективных управленческих решений – М., 2020. – 28 с.

6. Кузьмин А. И. Оценка программ: методология и практика. / Под ред. А. И. Кузьмина, Р. О'Салливан, Н. А. Кошелевой // М.: Издательство «Престо-РК». – 2009.

7. Кузьмин А. И. Глокальные проблемы в развитии оценки (https://evaluationconsulting. blogspot.com/2020/10/blog-post.html).

8. Кузьмин А. И., Кошелева Н. А. Теория изменений: общие рекомендации к применению (из опыта БДФ «Виктория») // М.: Изд-во «Проспект. – 2014.

9. Рождественская Н. В., Богуславская С. Б., Боброва О. С. Оценка эффективности проектов некоммерческих организаций, социального предпринимательства и гражданских инициатив [Книга]. – СПб.: Издательство Политехнического университета, 2016.

10. Стандарт доказательности практик в сфере детства (Версия № 2. 31 августа 2018) [В Интернете]. – http://www.ozenka. info/about/dokazatelniy_podhod/standart_dokazatelnosti_praktik/402_file_2.pdf.

ЗАКЛЮЧЕНИЕ

В данном обзоре проведен очень краткий анализ ряда ключевых терминов, входящих в языковой корпус предметной области, связанной с оценкой социальных проектов и программ. Надеемся, что представленный анализ будет способствовать выработке общих подходов и единого понятийного аппарата в области импакт-инвестиций и содействовать как формированию стандартов, так и более широкому внедрению в практику системы оценки благотворительных проектов и социального воздействия в целом.

Impact as an Object of Evaluation: Basic Concepts^[2]

In 2019, Russia approved the Concept for Promoting the Development of Charitable Activities for the Period up to 2025, which provides for the inclusion of social impact evaluation as a necessary stage in the implementation of charitable programs and projects. Thus, today the evaluation of social and economic impact is becoming an indispensable element of social projects.

Natalia Gladkikh,

Ph.D. in Psychology, Leading Expert of the Institute of Socio-Economic Design, Higher School of Economics

Zoya Talitskaya,

Expert, Gladway Foundation for the Development of Media Projects and Social Programs

The development of methodology and practical application of evaluation tools in the field of social projects and programs in Russia began more than 15 years ago. Since then quite a number of Russian-language manuals and methodological manuals for social projects evaluation have emerged created, digests of international publications on this topic have been published, several specialized organizations engaged in the evaluation of projects and programs have begun working, and the Association of Evaluation Specialists (AES) forming a community of specialists in the field of evaluation. In general, evaluation of projects and programs in Russia is an actively developing field of scientific and practical activity, and it has acquired a fairly long history and valuable experience.

An indicator of such experience is the thesaurus that has been formed for this subject area. It is available at least within the professional community of evaluation specialists. For example, the terms as “the Theory of Change”, “the Project Logical Model”, “the separation of categories of results and the impact of projects and programs” are established and shared by most specialists.

However, a number of difficulties is associated with defining and understanding basic terms. This is largely due to the fact that the language corpus of this branch of knowledge was originally borrowed from English and is often used as a language calque. For example, the terms “impact” and “assessment” are often used as direct transfer from English, and their definitions therefore require closer attention.

Let’s try to conduct a comparative analysis of the key terms included in the language corpus of the subject area of projects and programs evaluation.

BASIC CONCEPTS USED IN SOCIAL PROJECTS EVALUATION

The basic concept used in evaluating is expected to be the term “Evaluation" used in Russia in the direct meaning of its translation: “determination of value". One of the most popular and basic definitions of evaluation is the following definition proposed by A. I. Kuzmin [Kuzmin, 2009], based on the translation of a dictionary article by Michael Patton in 1997: “Program evaluation is a systematic collection of information about the activities of a program, its characteristics and results, which is carried out in order to make a judgment about the program, increase the effectiveness of the program and/or develop plans for the future." Note that in the context of this definition, the terms “program" and “project" could be considered synonymous, and thus it can be assigned to both categories.

Another common concept is “Monitoring", the meaning of which is also quite clearly defined in the Russian literature. Referring to one of the most classical definitions, also proposed by A. I. Kuzmin [Kuzmin, 2009], it is possible to define monitoring as “one of the types of program evaluation, which consists in constantly monitoring the progress of the program and comparing it with the plan… Monitoring of in-depth analysis does not imply, but provides operational information about the current status on the basis of comparing the actual values of a number of key figures (indicators) with their planned values."

The concept of “Assessment" is no less important, but the matter of its translation is more complicated. If you look at the second definition of this concept in the dictionary, you can designate its content as “assessment, determination of value", which does not give us grounds to determine its specificity in comparison with the previous one. However, the third definition is “definition, assessment (of viability)" – still allows you to determine this specificity. So, the main difference between this concept and the concept of “Evaluation" is the predictive nature of assessment – conditionally, you can characterize it as a “predictive" or “predictive" assessment, i.e. assessment of the present activity in the context of forecasting the development of this activity in the future, as well as forecasting the results of this activity. It is important to note here that for the second case there is an even more specific concept of “Impact Assessment", which can be translated into Russian as “прогнозирование влияния" (impact forecasting).

The International Association for Impact Assessment (IAIA) defines “Impact Assessment" as “the process of determining the future effects of current activities." As one of the modern examples of this term definition can be given the following: The Social Impact Assessment is “the process of preliminary assessment of the social consequences of certain social or political projects. It is a predictive procedure for assessing social effect" [Takyi, 2014].

The main difference between this concept and the concept of “Evaluation” is the predictive nature of assessment – conditionally, you can characterize it as a “predictive” or “predictive” assessment, i.e. assessment of the present activity in the context of forecasting the development of this activity in the future.

Thus, we can say that the term “assessment" (“assessment of influence"; predictive assessment) should be used in a situation when assessment is conducted even before the program launch, when we can determine the possible consequences and effects of our program, “monitoring" – when we assess the program compliance by a number of indicators in its implementation, with the possibility of prompt intervention. The assessment that is conducted after the program implementation is traditionally considered as the actual “program assessment" (“program impact assessment").

The key issues in this context are everything related to the definition of “outcome" and “impact" – social impact and the project impact, respectively. In many ways, the definition of the concept of social impact (effect) depends on the approach to its measurement. It can be noted that often the concept of social impact is considered as synonymous with the concept of “efficiency", as well as “social value".

The concept of social value, in turn, combines four components: the common good (for key stakeholders), the services provided, personal well-being (emotional well-being, increased self-evaluation, finding purpose), and community development [Courtney, 2018]. Different definitions also can be given to “Social efficiency". Thus, it can be defined as the ability to achieve the goal, the satisfaction of all stakeholders' interests, the optimal ratio of inputs and outputs, uninterrupted provision of resources, compliance with international standards [N. V. Rozhdestvenskaya, S. B. Boguslavskaya, O. S. Bobrova, 2016].

“Social impact" can be defined as referring to a wide range of non-economic outcomes affecting people's well-being, social capital, community development, and the environment. From the perspective of donors and investors, the social effect can be defined as the result of investing funds, which is determined not only quantitatively, but also qualitatively – in the form of new practices, knowledge and information.

Full understanding of the impact concept requires a shift from the expected activity results (outputs) to the expected final results (outcomes). Outputs are the results that organizations can directly measure or evaluate (e.g., number of trained people, percentage of new employees in the workforce), while outcomes are broader changes, benefits, and knowledge that can be gained in the medium to long term (e.g., reducing social exclusion, reducing inequality).

Since outcomes refer to changes in society, they are determined by the impact of a wide audience, as well as the influence of external conditions that could contribute to them. Organizations can play a key role in driving change, but their contributions should not be overstated. For this reason, the concept of social impact refers to that proportion of the total result that has been achieved as a result of the organization activities, in addition to what would have happened in any case [Grieco, 2015].

An example of possible areas for measuring social impact is given in the following table [Ibid.] (Table 1).

Table 1. Areas of possible measurements of social impact (according to Greco)

Figure 1. Chain of project results (according to A. I. Kuzmin)

Another important aspect of determining immediate results is to separate them from the characteristics of the activities performed. A. I. Kuzmin draws the attention on this aspect in one of his publications in the professional blog. The author provides the following diagram to illustrate this thesis.

Kuzmin notes that “in social design, outputs are the direct consequences of a series of actions within a project. It would seem that everything is simple: if you performed an action, you got an output. However, strange as it may seem, in both foreign and Russian publications, opinions on what should be considered an immediate result (output) differ radically. And it seems that our erroneous position is becoming more and more popular. It consists in the fact that the characteristics of the activity performed are referred to the immediate results. The author gives the following example: “Suppose a project is training. Training itself is a project activity. Nobody has any questions here. Now attention! The immediate results of training are called the following: how many training courses were organized, how many hours the classes lasted, how many people took part in the training. But all of the above refers to the activity itself, describes its scale, and answers the question of how much we have done. All these are characteristics (indicators) of the activity performed, but in no way its immediate results. What is the immediate result of training, its direct consequence? What happens to the trainees: changes in their knowledge, skills, and attitudes towards something?"

It is worth noting that the above-described problem of interpreting the concept of immediate results in terms of activity characteristics is quite common. Moreover, we can say that most of the sources considered during this analysis state that immediate results are understood exactly as activity characteristics.

Thus, the recommended position on this issue is the one that takes into account, on the one hand, the need to assess the results at the level of correlation with the stated objectives of the project, but on the other hand includes a component of the scale of these results, and therefore, an indication of the characteristics of the activity itself (the number of services provided, the audience covered, etc.).

If we try to analyze the present day Russian representations of key concepts related to valuation, it is worth quoting the position of experts from one of the associations of evaluation market players actively working towards the development of systemic solutions in the field of state programs and projects – the Accounts Chamber of the Russian Federation, as well as the Center for Advanced Management Decisions. In the manual on the theory of change, prepared in partnership by experts of these organizations, the following definitions are given [A. I. Kim et al., 2020]:

• immediate results (outputs) – specific products formed as a result of direct state influence and possible for use by target audiences (charity recipients);

• response (react) – the reaction of charity recipients to the use of immediate results (including in the form of behavior change);

• final results (outcomes) – a set of significant changes that the charity recipients experience after the use of immediate results;

• final impacts – desired medium- and long-term social and economic changes.

Outputs are the results that organizations can directly measure or evaluate, while outcomes are broader changes, benefits, and knowledge that can be gained in the medium to long term.

We cannot but mention in this context the approach of the Global Impact Investing Network (GIIN) in connection with the publication by this organization in November 2020 of a new evaluation methodology, presented as the first of its kind recommendation that allows the use of unified approaches and comparison of any projects with each other (Methodology for Standardizing and Comparing Impact Performance). It is worth noting, however, that this methodology is applicable to use by social investors, in order to select investment projects. Within this methodological guide, it is proposed to distinguish the following levels of evaluation: “outputs – short term outcomes – long terms outcomes". Outcomes analysis within this methodology should be built at several levels:

• Scale: number/coverage of stakeholders who received a result (outcome).

• Depth of impact: the degree of change obtained by stakeholders.

• Exposure time: the period of time during which the stakeholder continues to experience the result.

• Volatility: the degree of variability (change) of results over time.

Table 2. Basic terms and definitions related to impact and evaluation

It is worth paying attention to the removal of the “impact" category beyond the limits of the impact chain evaluation in the GIIN approach described above. We can say that such a position is common and represents a trend in the modern sense of the term. This, in particular, is noted by one of the leading international experts in the field of academic research for project and program evaluation, the author of the concept of 4 Strategies, Alnur Ebrahim [Ebrahim, 2019].

Thus, if we proceed from the global modern trends in this concept definition, we can say that the impact concept is becoming more and more “common noun", defining in general the whole range of concepts related to the project results description, with the exception of only immediate results. We also recommend using this understanding in the language corpora to develop a model for evaluation the social and economic impact of social projects. This is the definition used in this paper.

THE THEORY OF CHANGES AS THE BASIC PROBABLE COMPONENT OF THE MODEL FOR SOCIAL PROJECTS ASSESSMENT

One of the most researched and described in the scientific literature basic constructs and bases of evaluation is the concept of the Theory of Change (ToC) of the project, which is a clear and precise description of the alleged relationships between the causes (actions) and the expected effects (results) that are planned to be achieved under project, indicating the conditions for achieving each of the levels of results. The Theory of Change explains the reasons why certain approaches to solving the problem are used, as well as how these approaches will lead to the desired changes.

As noted by A. I. Kuzmin, the author of the first Russian-language publication on the Theory of Changes, ToC can be used as a tool for solving a whole range of problems [A. I. Kuzmin, N. A. Kosheleva, 2014]:

• As a project results describing tool;

• Design tool (project design intent development);

• A way to visualize the project intent;

• A way to justify the causal relationship between project activities and their results;

• A way to define project impact boundaries;

• Basis for the formation of a project monitoring system;

• Basis for project evaluation;

• A way to “archive" good practices in a generalized form.

Traditionally and in its most general form, the Theory of Change is most often presented in the form of a map of expected results, showing how the actions under project will lead to the achievement of the project goal, and a list of assumptions or conditions under which this “theory" will work, which the authors of the project expect to comply with and which, at the same time, are beyond their influence.

The Theory of Change is one of the most common tools used to describe the social technology of a project, for the purpose of its subsequent evaluation. In other words terminology wise, it can be attributed specifically to the field of projects and programs evaluation. It has been actively studied in this area. So, you can find various classifications of types of ToC, approaches to its development, including specialized software, author developed approaches to assessment based on ToC, etc.

The Theory of Change development can be called one of the universal tools used in project evaluation.

EVIDENCE-BASED APPROACH IN SOCIAL PROJECTS ASSESSMENT

Another fundamental term is closely related to the Theory of Change theme. It is the evidence based approach of the social technology applied in a particular case. Generally speaking, the evidence-based approach is an approach based on the application of research methods, on the search and study of arguments that confirm that the set of actions carried out within approach does indeed lead to the desired change.

The evidence-based approach to social design is actively developed and implemented nowadays. This is evidenced by both the increase in the number of scientific publications and outside academic discussions and even the creation of educational programs in line with the evidence-based approach (an example of such initiative is the master’s program Evidence-Based Design and Evaluation of Programs in Social Risk Management in Childhood, opened five years ago at MSUPE (Moscow State University of Psychology and Education)).

As a justification of importance of evidence-based approach in the social sphere, we can quote two expert heuristics: illusions of “pure field" and “absolute wealth". The first is when starting a social project and proposing a new (own) solution to a social problem, authors often ignore the existing experience in solving this problem. Such a heuristic is impossible in a case, for example of a commercial project, where an important condition for its effectiveness is taking into account the market context. But it is quite possible in the public sector and the likelihood of its occurrence is high. Market research, analysis of the target audience needs, highlighting of the leading players and existing technologies, analysis of the effectiveness of existing solutions are often not considered as mandatory when developing a social project.

The Theory of Change is most often presented in the form of a map of expected results, showing how the actions under project will lead to the achievement of the project goal.

The essence of the second heuristic is that “whatever social service we provide it is definitely a good and right thing to do." Hence the widespread, humorous concept of “doing good" arose. If the service is not of high quality, does not correspond to modern ideas (including scientific ones) regarding reasons and possible approaches to solving a social problem, etc., it cannot be said that the project really contributes to the solution of this problem.

This issue becomes especially acute in a situation where the project charity recipients are vulnerable groups who, for various reasons, cannot express their opinion regarding the quality of the social services provided to them and their general attitude to the project intervention. These are, in particular, children, people with mental disabilities, homeless people, etc. In this regard, it is reasonable that the topic of practice certification in Russia has been developed in the context of children’s programs. It is worth considering ways of its possible development and application of this cited standard as an example [Practice certification standard in the field of childhood (Version No. 2. 31 August, 2018)].

REFERENCES

1. Courtney Paul Conceptualising Social Value for the Third Sector and Developing Methods for Its Assessment [Magazine]. – [s.l.]: International Society for Third-Sector Research Voluntas, 2018 – https://doi.org/10.1007/s11266–017–9908–3: Vol. 29.

2. Ebrahim Alnoor Measuring Social Change: Performance and Accountability in a Complex World – [Book]. – 2019.

3. Grieco Cecilia Assessing Social Impact of Social Enterprises. Does One Size Really Fit All? [Book]. – [s.l.]: Springer, 2015.

4. Takyi Stephen Appiah REVIEW OF SOCIAL IMPACTS ASSESSMENT (SIA): APPROACH, IMPORTANCE, CHALLENGES AND POLICY IMPLICATIONS [Magazine]. – [s.l.]: International Journal of Arts & Sciences, 2014 – Vl. 07(05):217–234 (2014).

2. The article uses the materials of the research report "Research of modern experience of project activities of non-profit organizations evaluation", prepared under the guidance of the leading expert of the Institute of Socio-Economic Design, National Research University Higher School of Economics, and Ph.D. in Psychology N.Yu. Gladkikh.