Applying the criteria indicators and metrics evaluation method on ICT research: The HUMBOLDT project

Evaluating an Information Technology (IT) based research and development project from an end-user's perspective is necessary in order to discover whether or not the outcomes of the project are successful. However, it is not straightforward to design and implement a generic user evaluation methodology that can be applied to a large-scale collaborative integrated research and development project. In this article, we share our experience based on the research evaluation carried out in a large-scale integrated ICT research project-The HUMBOLDT project. The project develops IT tools and services for cross-border geospatial data harmonization, which are applied to nine different application scenarios. A novel user-oriented evaluation methodology, namely-Criteria Indicators and Metrics (CIM)-is defined and utilized for the evaluation of the project research outcomes. The user evaluation results reveal the extent to which the research and development objectives of the project are achieved, and also demonstrate the benefits and limitations of the evaluation methodology. The results of this article can be usefully applied to other collaborative research projects in a variety of domains.