Empirically-based method for performing qualitative synthesis in software engineering

Context: In recent years, software researchers begun to benefit from qualitative synthesis to systematically build knowledge about the impact and preconditions of software technologies. Qualitative synthesis refers to qualitative research methods (as opposed to quantitative) used to bring together empirical evidence in a narrative or systematic review. However, only few of the qualitative syntheses published in software engineering have shown a rigorous approach to qualitative synthesis. Thus, the current qualitative synthesis findings are neither understandable nor trustworthy. Problem: These results indicate software researchers have difficulties in identifying quality issues in qualitative syntheses and in applying existing qualitative synthesis methods. I found three problems that have hindered the use of qualitative synthesis methods in software engineering. First, the lack of consensus on the meaning on quality has impeded the definition of quality criteria and the design of a quality appraisal instrument. Second, it is unclear to what extent qualitative synthesis methods are suitable for synthesizing evidence in software engineering. Finally, the lack of evidence on the suitability of qualitative synthesis methods has impeded their proper tailoring to software engineering needs and so, their effective use. Objectives: I aimed in this PhD thesis to design a reliable and valid quality appraisal instrument for qualitative synthesis; empirically evaluate the suitability of the most mature qualitative synthesis methods, i.e., meta-ethnography and thematic analysis; and enhance existing guidelines for meta-ethnography and thematic analysis. Contributions: First, I developed an instrument to support software researchers in appraising the quality of qualitative synthesis using psychometric guidelines. I operationalized the notion of quality of qualitative synthesis into four criteria: inter-subject comprehensibility, indication of the research process, trustworthiness, and relevance. Then, I designed and validated the instrument. The results of two empirical studies indicate the instrument has acceptable inter-rater agreement, substantial inter-rater reliability, good criterion validity, and high acceptance. Second, I designed and performed a multiple-case study to characterize the suitability of meta-ethnography and thematic analysis in software engineering. Third, I tailored the current guidelines for performing meta-ethnography and thematic analysis. I specified a prescriptive process; formalize the identification, extraction, and integration of evidence; and formalize the notation used for documenting synthesis findings. Then, I designed and performed interviews to validate the proposed guidelines. The results indicates software researchers perceived the proposed guidelines as understandable, useful, and easy to use.