Introduction
⌅The challenge of measuring the impact of scientific research conducted within the framework of R&D&I projects is a novel issue that necessitates standardizing patterns to predict, design, and measure the short-, medium, and long term effects of its results (Estébanez, 1998ESTÉBANEZ, M.E.: “La medición del impacto de la ciencia y la tecnología en el desarrollo social”, En: Taller de Indicadores de Impacto Social de la Ciencia y la Tecnología, RICYT, La Cumbre, Argentina, La Cumbre, Argentina, 1998.; 2002ESTÉBANEZ, M.E.: “Impacto social de la ciencia y la tecnología: estrategias para su análisis”, En: RICYT: El estado de la ciencia. Principales indicadores deficiencia y tecnología iberoamericanos/interamericanos, Argentina, pp. 95-103, 2002.; Itzcovitz & Fernández, 1998ITZCOVITZ, V.; FERNÁNDEZ, P.E.: “Propuesta metodológica sobre la medición del impacto de la C y T sobre el desarrollo social”, En: Segundo Taller de Indicadores de Impacto Social de la Ciencia y la Tecnología, RICYT, Ed. Red Internacional de Indicadores RICYT, La Cumbre, Argentina, 1998.; Coria, 2008CORIA, I.D.: “El estudio de impacto ambiental: características y metodologías”, Invenio, 11(20): 125-135, 2008, ISSN: 0329-3475.; Guerra Castillo, 2021GUERRA CASTILLO, S.: “Metodología para evaluar el impacto de la capacitación en las empresas cubanas”, Revista Universidad y Sociedad, 13(6): 237-249, 2021, ISSN: 2218-3620.). Predicting, designing, and measuring the impact of scientific research is a crucial topic today, as its implementation leads to a series of decisions regarding science policy and the direction of research work for the benefit of society (CITMA-Cuba, 2012CITMA-CUBA: Reglamento para el proceso de elaboración, aprobación, planificación, ejecución y control de los Programas y Proyectos de Ciencia, Tecnología e Innovación, Resolución 44 /2012 del CITMA, Inst. Ministerio de Ciencia, Tecnología y Medio Ambiente, Rresolución 44/2012, La Habana, Cuba, 2012.; MES-Cuba, 2012MES-CUBA: Objetivos de trabajo para el año 2013 y hasta el 2016, Inst. Ministerio de Educación Superior (MES), informe institucional, La Habana, Cuba, 2012.; Alonso et al., 2019ALONSO, B.I.; GORINA, S.A.; PÉREZ, S.R.M.; FIGUEROA, R.J.: “Evaluación de la pertinencia e impacto científico de los resultados investigativos en universidades cubanas”, Revista Universidad y Sociedad, 11(4): 325-334, 2019, ISSN: 2218-3620.).
The system of indicators used in this study to predict, design, and measure the impact of research projects is a tool that allows professors and researchers, from the project design stage, to anticipate certain aspects that, when considered from the project's inception, will contribute to measuring the impact of the results. This study considered the necessary link between the results obtained in universities and research centers through projects and the results that can be transferred to local institutions, impacting local development. Documentary research techniques were used to establish indicators, designed for four dimensions (scientific, economic, social, and environmental), with 30 indicators. The system was theoretically validated using the Delphi method of expert consultation, considered one of the most reliable subjective forecasting methods, as it combines subjective analysis criteria with mathematical-statistical analysis of the results (Oñate et al., 1988OÑATE, N.; RAMOS, L.; DIAZ, A.: “Utilización del Método Delphi en la pronosticación: Una experiencia inicial”, Cuba: Economía Planificada, 3(4): 9-48, 1988.; Estévez & Gallastegui, 2005ESTÉVEZ, M. de L.B.; GALLASTEGUI, J.J.A.: “El método Delphi. Su implementación en una estrategia didáctica para la enseñanza de las demostraciones geométricas”, Revista Iberoamericana de educación, 36(7): 1-10, 2005, ISSN: 1681-5653.; Cruz, 2007CRUZ, M.: “El Método Delphi en las Investigaciones”, En: Congreso PEDAGOGÍA’2007, La Habana, Cuba, p. 40, 2007.; Blasco et al., 2010BLASCO, M.J.E.; LÓPEZ, P.A.; MENGUAL, A.S.: “Validación mediante el método Delphi de un cuestionario para conocer las experiencias e interés hacia las actividades acuáticas con especial atención al Winsurf”, 2010, ISSN: 1578-2174.; Fernández de Castro & López, 2013FERNÁNDEZ DE CASTRO, F.A.: Los proyectos de investigación y su impacto. Validación y propuesta de un sistema de indicadores, Inst. Centro Nacional de Derecho de Autor (CENDA), Certificación de depósito legal facultativo de obras protegidas del CENDA, Registro: 2233-2013, La Habana, Cuba, 2013.).
The implementation of this indicator system for predicting, designing, and measuring the impact of universities and research centers, carried out in 10 higher education institutions across the country, served as the basis for applying the IADOV technique. This technique is recognized for evaluating process satisfaction and allowed us to determine the level of user satisfaction with the implementation of the indicator system (López & González, 2002LÓPEZ, A.; GONZÁLEZ, V.: “La técnica de Iadov. Una aplicación para el estudio de la satisfacción de los alumnos por las clases de educación física”, Revista Digital-Buenos Aires, 8(47), 2002.; Fernández de Castro et al., 2020bFERNÁNDEZ DE CASTRO, F.A.; ORTEGA, N.S.; FARRAT, Y.R.: “El proceso de validación mediante la Técnica de Iadov en cursos por encuentros”, Ingeniería Agrícola, 10(1): 66-70, 2020b, ISSN: 2306-1545.). As a result, the feasibility of its use was confirmed, expressed quantitatively in the high Group Satisfaction Index and qualitatively in the feedback provided, which demonstrates satisfaction with the contribution of the indicator system. This reflects acceptance of the proposal and recognition of its usefulness. This indicator system is of great importance in meeting the information needs of decision-makers and contributes to strengthening local management, serving as a very useful tool for professors and researchers developing research projects (Hurtado de Mendoza & Méndez, 2007HURTADO DE MENDOZA, S.; MÉNDEZ, D.: Software educativo, Método de Consulta a Expertos (Delphi), (Versión Delphi. CECOFIS), La Habana, Cuba, 2007.; De las Cuevas et al., 2020DE LAS CUEVAS, H.R.; GÓMEZ, R.I.; FERNÁNDEZ DE CASTRO, F.A.; PANEQUE, R.P.: “Software para la evaluación del impacto de los proyectos de investigación en Ingeniería Agrícola”, Ingeniería Agrícola, 10(3), 2020, ISSN: 2227-8761.).
To facilitate the use of the indicators, software was designed that allows users to input the data necessary to determine which indicators to use when developing a project to achieve impact. It also allows for the evaluation of the impact of completed projects and the comparison of results with those obtained at different project stages (De las Cuevas et al., 2020DE LAS CUEVAS, H.R.; GÓMEZ, R.I.; FERNÁNDEZ DE CASTRO, F.A.; PANEQUE, R.P.: “Software para la evaluación del impacto de los proyectos de investigación en Ingeniería Agrícola”, Ingeniería Agrícola, 10(3), 2020, ISSN: 2227-8761.).
Development
⌅Impact assessment uses both quantitative and qualitative methods, which are not mutually exclusive and are typically used in combination.
For comparison, an analysis of compliance with the use of indicators is conducted through the intervention of completed research projects. The impact of objectives can also be measured in the short, medium, and long term.
The indicator system is a tool for predicting, designing, and measuring the impact of research projects. It is based on the definition of different dimensions and the indicators that comprise them, and is designed to measure in what area and to what extent a research project achieves or will achieve impact. Its objective is to provide researchers, professors, and technicians at research centers, universities, and other entities related to research projects with an easy-to-use tool that allows for the assessment of impacts through the use of indicators at different stages of scientific research projects. In other words, it allows for the estimated quantification of the lasting change expected to be produced in the scientific, technological, economic, social, and environmental spheres.
The implementation of this indicator system is planned in 5 steps:
-
Analysis of at least 10 completed research projects for design analysis. Partial and final reports.
-
Adaptation of the indicators, using the Iadov technique (López y González, 2002LÓPEZ, A.; GONZÁLEZ, V.: “La técnica de Iadov. Una aplicación para el estudio de la satisfacción de los alumnos por las clases de educación física”, Revista Digital-Buenos Aires, 8(47), 2002.).
-
Introduction of improvements to the indicator system based on the results and experiences obtained. Consultation with experts, using the Delphi method (Blasco et al., 2010BLASCO, M.J.E.; LÓPEZ, P.A.; MENGUAL, A.S.: “Validación mediante el método Delphi de un cuestionario para conocer las experiencias e interés hacia las actividades acuáticas con especial atención al Winsurf”, 2010, ISSN: 1578-2174.).
-
Proposal for the evaluation of 10 new projects as part of the planning process to determine their impact through their design.
-
Use of the IMPACT Software as a comparative method (De las Cuevas et al., 2020DE LAS CUEVAS, H.R.; GÓMEZ, R.I.; FERNÁNDEZ DE CASTRO, F.A.; PANEQUE, R.P.: “Software para la evaluación del impacto de los proyectos de investigación en Ingeniería Agrícola”, Ingeniería Agrícola, 10(3), 2020, ISSN: 2227-8761.).
The validated indicator system can be used efficiently during the execution of a multi-year project, and its impact can be evaluated over the different years of project implementation. As a guide, method, or procedure for the qualitative and quantitative analysis and evaluation of the results, we will use the system of indicators for predicting, designing, and measuring the impact of research projects. This system provides 30 indicators across four dimensions (scientific, economic, social, and environmental), as cited in the bibliography (Fernández de Castro et al., 2020aFERNÁNDEZ DE CASTRO, F.A.; LÓPEZ, P.A.; GÓMEZ, A.M.V.: “Sistema de indicadores para prever, diseñar y medir el impacto de los proyectos de investigación”, Ingeniería Agrícola, 10(2): 50-58, 2020a, ISSN: 2306-1545.) and the IMPACTp software (De las Cuevas et al., 2020DE LAS CUEVAS, H.R.; GÓMEZ, R.I.; FERNÁNDEZ DE CASTRO, F.A.; PANEQUE, R.P.: “Software para la evaluación del impacto de los proyectos de investigación en Ingeniería Agrícola”, Ingeniería Agrícola, 10(3), 2020, ISSN: 2227-8761.).
IMPACTp Software
⌅The software for evaluating the impact of research projects, called “IMPACTOp,” is a tool for assessing the impact of research projects based on a system of indicators proposed by Fernández de Castro (2013)FERNÁNDEZ DE CASTRO, F.A.: Los proyectos de investigación y su impacto. Validación y propuesta de un sistema de indicadores, Inst. Centro Nacional de Derecho de Autor (CENDA), Certificación de depósito legal facultativo de obras protegidas del CENDA, Registro: 2233-2013, La Habana, Cuba, 2013.; Fernández de Castro & López (2013)FERNÁNDEZ DE CASTRO, F.A.; LÓPEZ, P.A.: “Validación mediante método Delphi de un sistema de indicadores para prever, diseñar y medir el impacto sobre el desarrollo local de los proyectos de investigación en el sector agropecuario”, Revista Ciencias Técnicas Agropecuarias, 22(3): 54-60, 2013, ISSN: 2071-0054.. It is structured around four dimensions: knowledge, economic, social, and environmental, and includes 30 indicators designed to measure in which areas and to what extent a research project achieves or will achieve impact. This system automates the estimated quantification of the change expected to occur in the scientific, economic, social, and environmental spheres. The software design involved a theoretical analysis of the impacts that the project will generate in the different areas or spheres within its context. This paper presents an impact analysis using a rating method for each indicator on a scale of 1 to 5 points, with the values subsequently normalized to a scale of 1 to 100. The “IMPACTp” software allows for the rapid determination of the main indicators that characterize the Knowledge, Economic, Social, and Environmental dimensions. in research projects. “IMPACTp” is easy to use, with each part accessible through the user's link to the Main Control Panel.
A case study was recently conducted, with the preliminary evaluation of the first two semesters of the project's first year: Agricultural Technologies for the Resilience and Sustainability of Cuban Agroecosystems (Code: PN131LH001.85).
The project's impact assessment was carried out using the indicator system to predict, design, and measure the impact of research projects that demonstrate the impact achieved in the first year of the project (2024), in addition to the case study of the UNAH subproject with the results obtained on the 5 selected farms. As a procedure for the qualitative and quantitative analysis and evaluation of the results, the indicator system was used to predict, design, and measure the impact of research projects, which has 4 dimensions (scientific, economic, social, and environmental) and 30 indicators, cited in the bibliography. (Fernández de Castro et al., 2020bFERNÁNDEZ DE CASTRO, F.A.; ORTEGA, N.S.; FARRAT, Y.R.: “El proceso de validación mediante la Técnica de Iadov en cursos por encuentros”, Ingeniería Agrícola, 10(1): 66-70, 2020b, ISSN: 2306-1545.), it was found in this preliminary evaluation, with the use of the system of indicators, that it met the indicators of the impact of knowledge, economic, social and environmental.
Analysis of the Results
⌅At least 100 producers (20% women) from the agricultural production sector were trained. Training workshops on various topics were conducted, with 300 participants, 73 of whom were women (24%). A total of 12 field experiments were carried out on different agricultural crops. Six experiments are currently underway. Post-harvest handling was evaluated. Studies on food loss and waste were conducted, as well as an assessment of the processing and post-harvest handling of medicinal plants.
Publications were produced in all four research groups and in documents that explain the new or improved knowledge obtained and incorporate it into the innovation cycle of local entities. Doctoral dissertations, Master's and Specialization theses, and Diploma projects are in progress, all of which contribute to solving real local problems. The project participated in Science and Technology Forums. Awards have been received; the project should be recognized for the quality of its scientific contribution and its practical application. Science and Technology Forum Awards, Provincial Innovation Awards, and Branch Awards. Participation in relevant scientific events in Cuba and abroad. A significant number of professors and researchers participated in the project, as well as students from student research groups linked to the project. It has a multidisciplinary character, integrating various disciplines.
Social Impact. Solutions to local development problems: proposals based on scientific knowledge for solving local problems. Engagement with social actors: establishing links with local and external social actors, such as the local government, municipal and provincial CITMA (Ministry of Science, Technology and Environment), NGOs, universities, and research centers (a network of actors contributing to a local innovation system), necessary to channel the relationship between the results and the local community. Use of participatory methods: the basis of communication between researchers and social actors.
Training: 100 people were trained appropriately for the individuals, communities, and local institutions responsible for continuing the activities after the projects are completed. Local institutions linked to the project: for the practical implementation of the contributions, which will subsequently validate their importance and the benefits obtained.
Intellectual Property Proposals. Protection should be provided for through both industrial property (patents, plant variety registration, or other forms) and copyright (software, multimedia, web). If the project involves other institutions, specify their percentage of participation with respect to the results eligible for protection.
Participation in scientific events, presentation of papers at relevant scientific events in Cuba and abroad. Number of professors and researchers participating in the project. Number of students from student research groups linked to the project. The multidisciplinary nature of the projects to be evaluated: integration of different disciplines. Amount of contributions to knowledge resulting from the research. Regarding the protection of the results, the next task will be managed, as appropriate, through Intellectual Property rights, specifically copyright, with the registration and optional legal deposit of protected works, theses, reports, and other generated documents that are eligible for protection, before the National Copyright Center (CENDA). As a procedure for the qualitative and quantitative analysis and evaluation of the results, the system of indicators for predicting, designing, and measuring the impact of research projects was used. This system, with its four dimensions (scientific, economic, social, and environmental) and 30 indicators, is cited in the bibliography (Fernández de Castro et al., 2020bFERNÁNDEZ DE CASTRO, F.A.; ORTEGA, N.S.; FARRAT, Y.R.: “El proceso de validación mediante la Técnica de Iadov en cursos por encuentros”, Ingeniería Agrícola, 10(1): 66-70, 2020b, ISSN: 2306-1545.). Initially, through a review of the partial reports and the final report for 2024, the identification and analysis of the indicators' behavior during this period was carried out. This preliminary evaluation confirmed that the project had met 80% of the knowledge impact indicators and over 90% of the economic, social, and environmental indicators. It should be noted that, as a result of this preliminary analysis, it was decided to include one new indicator in the system, namely the one related to women's inclusion in the social dimension.
Subsequently, the data was entered into the IMPACTp software, version 01, for evaluation. The index obtained using the IMPACTp software allowed for a numerical comparison of the results. The overall evaluation result was 93.33 (Figure 1).
Regarding the evaluation of the knowledge dimension, the intellectual property indicator negatively impacted the overall result for this dimension.
Regarding the protection of the results, this indicator will be managed as appropriate, through Intellectual Property rights, specifically copyright, with the optional registration and legal deposit of protected works, theses, reports, and other documents generated that are eligible for protection, before the National Copyright Center (CENDA).
These analyses served to compare the partial results and will serve to compare them with the next stages, thus allowing for ongoing monitoring of the project. In the first half of 2025, work continued on the project's impact evaluation using the indicator system to predict, design, and measure the impact of the research projects, demonstrating the impact achieved compared to that obtained in the first year of the project, 2024.
Conclusions
⌅These comparative analyses of the partial results of the research projects, using the indicator system for predicting, designing, and measuring the impact of research projects and the IMPACTp software, will serve to compare the results obtained with the next stages of the project. This will allow for tracking tasks and monitoring the partial results of the different stages.