RESUMEN
BACKGROUND: Conversational agents are currently a valid alternative to humans in first-level interviews with users who need information, even in-depth, about services or products. In application domains such as health care, this technology can become pervasive only if the perceived "quality in use" is appropriate. How to measure chatbot quality is an open question. The international standard ISO/IEC 25010 proposes a set of characteristics (effectiveness, efficiency, satisfaction, freedom from risk, and context coverage) to be considered when the "quality in use" of a software system has to be measured. BASIC PROCEDURE: This study proposes a clinical chatbot comparison method based on quality. The proposed approach is based on Analytic Hierarchy Process methodology (AHP). FINDINGS: Our contribution is twofold. First, we propose a set of measures for each characteristic of ISO/IEC 25010 according to three classes of functionality: providing information, providing prescriptions and process management. Moreover a quantitative method is proposed for making homogeneous the pairwise weights when the AHP is used for the "quality-in-use" comparison. As a case study, a comparison of two versions of a chatbot was performed. CONCLUSIONS: The results show that the proposed approach provides an effective reference base for performing quality comparisons of medical chatbots compliant with the ISO/IEC 25010 standard.