Elicitation of Rank Correlations with Probabilities of Concordance: Method and Application to Building Management.

Constructing Bayesian networks (BN) for practical applications presents significant challenges, especially in domains with limited empirical data available. In such situations, field experts are often consulted to estimate the model's parameters, for instance, rank correlations in Gaussian copula-based Bayesian networks (GCBN). Because there is no consensus on a 'best' approach for eliciting these correlations, this paper proposes a framework that uses probabilities of concordance for assessing dependence, and the dependence calibration score to aggregate experts' judgments. To demonstrate the relevance of our approach, the latter is implemented to populate a GCBN intended to estimate the condition of air handling units' components-a key challenge in building asset management. While the elicitation of concordance probabilities was well received by the questionnaire respondents, the analysis of the results reveals notable disparities in the experts' ability to quantify uncertainty. Moreover, the application of the dependence calibration aggregation method was hindered by the absence of relevant seed variables, thus failing to evaluate the participants' field expertise. All in all, while the authors do not recommend to use the current model in practice, this study suggests that concordance probabilities should be further explored as an alternative approach for the elicitation of dependence.

Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions.

INTRODUCTION: Research in various areas indicates that expert judgment can be highly inconsistent. However, expert judgment is indispensable in many contexts. In medical education, experts often function as examiners in rater-based assessments. Here, disagreement between examiners can have far-reaching consequences. The literature suggests that inconsistencies in ratings depend on the level of performance a to-be-evaluated candidate shows. This possibility has not been addressed deliberately and with appropriate statistical methods. By adopting the theoretical lens of ecological rationality, we evaluate if easily implementable strategies can enhance decision making in real-world assessment contexts. METHODS: We address two objectives. First, we investigate the dependence of rater-consistency on performance levels. We recorded videos of mock-exams and had examiners (N=10) evaluate four students' performances and compare inconsistencies in performance ratings between examiner-pairs using a bootstrapping procedure. Our second objective is to provide an approach that aids decision making by implementing simple heuristics. RESULTS: We found that discrepancies were largely a function of the level of performance the candidates showed. Lower performances were rated more inconsistently than excellent performances. Furthermore, our analyses indicated that the use of simple heuristics might improve decisions in examiner pairs. DISCUSSION: Inconsistencies in performance judgments continue to be a matter of concern, and we provide empirical evidence for them to be related to candidate performance. We discuss implications for research and the advantages of adopting the perspective of ecological rationality. We point to directions both for further research and for development of assessment practices.

Integrated assessment and mapping of provisioning services for sustainable management of natural resources, the case of Lake Hawassa Basin, Ethiopia.

Lake Hawassa Basin (LHB)-the study area is known for its rich and diverse aquatic and terrestrial natural resource base. However, the prevailing environmental and social problems, such as land degradation, deforestation, pollution, resource exploitation, etc. impacted the existing provisioning services (PS), and the effect becomes remarkable unless sound management is in place. The study aimed at the assessment and mapping of PS to suggest development options for decision-makers. The study employed various methods including primary and secondary data collection, including existing Land Use Land Cover (LULC), desk review, stakeholder consultations, site visits, expert judgment matrix, and ArcGIS v10.1. The study results include 6 PS identified and prioritized from the existing 14 PS, mapping of the spatial pattern of the selected 6 PS at the basin scale, and alternative development options recommended for the decision-making process conducted by decision-makers and development partners to ensure efficient management of ecosystem services in LHB. The importance of this study, as well as the simplicity and user-friendly nature of the methods and approach adopted, enables interested parties to replicate while conducting similar studies in different places within the country or globally. The intervention of adopting this study approach helps also to avoid or minimize the aforesaid biophysical and socioeconomic environmental problems and ensure development activities planned or implemented in the respective study area are environmentally friendly, and socially acceptable, through sustainable management of natural resources. In this regard, decision-makers and development partners shall provide adequate consideration for this study approach and the result of demonstrating basin scale spatial variability of PS. This plays a vital role in the sustainable management of natural resources as well as provisioning services existing in the study area to benefit the community members, ensure human well-being, and secure the livelihood of the people residing within or around the Lake Hawassa Basin.

When expert predictions fail.

We examine the opportunities and challenges of expert judgment in the social sciences, scrutinizing the way social scientists make predictions. While social scientists show above-chance accuracy in predicting laboratory-based phenomena, they often struggle to predict real-world societal changes. We argue that most causal models used in social sciences are oversimplified, confuse levels of analysis to which a model applies, misalign the nature of the model with the nature of the phenomena, and fail to consider factors beyond the scientist's pet theory. Taking cues from physical sciences and meteorology, we advocate an approach that integrates broad foundational models with context-specific time series data. We call for a shift in the social sciences towards more precise, daring predictions and greater intellectual humility.

Validez de las reglas de un análisis estadístico implicativo según juicio de expertos

Introducción: una nueva metodología para el empleo del análisis estadístico implicativo contextualizado a las investigaciones médicas de causalidad brinda reglas y meta-reglas que deben corresponderse con la práctica médica, lo cual se desconoce. Objetivo: evaluar la calidad de las reglas derivadas de la aplicación del análisis estadístico implicativo en las investigaciones médicas de causalidad. Métodos: se realizó un estudio observacional con la aplicación de técnicas cualitativas de tipo criterio de expertos. Los 33 expertos de mayor puntuación emitieron su juicio, mediante una escala Likert de cinco puntos, acerca de la frecuencia con que se cumplen en la práctica médica 23 reglas derivadas de la aplicación del análisis estadístico implicativo en tres estudios para la identificación de factores pronósticos en cáncer de mama. Las opiniones se procesaron mediante el método Delphi en dos rondas y se trianguló con la validez racional de estas reglas. Resultados: los expertos consideraron que cerca del 74 % de las reglas se cumplen en la práctica médica siempre o frecuentemente y sólo una regla resultó absurda. En la segunda ronda se trataron las tres reglas en las que no existió un acuerdo perfecto, en la cual se ratificó una regla como absurda y aumentó el nivel de acuerdo. La triangulación permitió encontrar contradicciones literatura-experto que facilitaron la identificación de problemas en la redacción de la interpretación de las reglas. Conclusiones: las reglas evaluadas reflejan, en general, lo que ocurre en la práctica médica y aquellas que resultan absurdas alertan al investigador sobre un posible sesgo de confusión.

Introduction: A new methodology for the use of contextualized statistical implicative analysis in causality medical research provides rules and meta-rules that must correspond to medical practice, which is unknown. Objective: To evaluate the quality of the rules derived from the application of the statistical implicative analysis in causality medical research. Methods: An observational study was carried out applying qualitative techniques based on expert judgement. Using a five-point Likert scale, the 33 experts with the highest score expressed their judgment about the frequency of the fulfilment of the 23 rules derived from the application of the statistical implicative analysis in three studies in order to identify prognostic factors for breast cancer. Opinions were processed using the two-round Delphi method, and the rational validity of these rules was triangulated. Results: The experts considered that about 74% of the rules are always or frequently followed in medical practice; only one rule was absurd. In the second round, the three rules in which there was no perfect agreement were discussed; one rule was ratified as absurd and there was an increase in the level of agreement. The triangulation allowed us to find contradictions between literature and experts, which facilitated the identification of problems in the drafting of the interpretation of the rules. Conclusions: The rules evaluated generally reflect what happens in medical practice and those that are absurd warn the researcher about the possibility of a confusion bias.

Application of a practice-based approach in variable selection for a prediction model development study of hospital-induced delirium.

BACKGROUND: Prognostic models of hospital-induced delirium, that include potential predisposing and precipitating factors, may be used to identify vulnerable patients and inform the implementation of tailored preventive interventions. It is recommended that, in prediction model development studies, candidate predictors are selected on the basis of existing knowledge, including knowledge from clinical practice. The purpose of this article is to describe the process of identifying and operationalizing candidate predictors of hospital-induced delirium for application in a prediction model development study using a practice-based approach. METHODS: This study is part of a larger, retrospective cohort study that is developing prognostic models of hospital-induced delirium for medical-surgical older adult patients using structured data from administrative and electronic health records. First, we conducted a review of the literature to identify clinical concepts that had been used as candidate predictors in prognostic model development-and-validation studies of hospital-induced delirium. Then, we consulted a multidisciplinary task force of nine members who independently judged whether each clinical concept was associated with hospital-induced delirium. Finally, we mapped the clinical concepts to the administrative and electronic health records and operationalized our candidate predictors. RESULTS: In the review of 34 studies, we identified 504 unique clinical concepts. Two-thirds of the clinical concepts (337/504) were used as candidate predictors only once. The most common clinical concepts included age (31/34), sex (29/34), and alcohol use (22/34). 96% of the clinical concepts (484/504) were judged to be associated with the development of hospital-induced delirium by at least two members of the task force. All of the task force members agreed that 47 or 9% of the 504 clinical concepts were associated with hospital-induced delirium. CONCLUSIONS: Heterogeneity among candidate predictors of hospital-induced delirium in the literature suggests a still evolving list of factors that contribute to the development of this complex phenomenon. We demonstrated a practice-based approach to variable selection for our model development study of hospital-induced delirium. Expert judgement of variables enabled us to categorize the variables based on the amount of agreement among the experts and plan for the development of different models, including an expert-model and data-driven model.

A simplified material flow analysis employing local expert judgment and its impact on uncertainty.

Material flow analysis (MFA) is an effective tool for waste management, but low- and middle-income countries lack essential data for MFA. This study proposed a simplified MFA (sMFA) utilizing local expert judgment (LEJ) and examining the impact of simplification on its uncertainty. A stochastic sMFA model was developed for nitrogen and phosphorus in urban Mandalay, Myanmar. This model was compared with the intensive MFA (iMFA) model employing intensive surveys for primary data collection. For the total loadings to the environment, the medians of the sMFA were higher by 3% and 11%, respectively, for nitrogen and phosphorus than those of the iMFA. The widths of the 80% confidence intervals of these loadings in the sMFA, normalized by those in the iMFA, were - 0.05 and - 0.11, respectively. The three largest flows to the environment were the same for the two models: on-site sanitation effluent/leakage, greywater, and industrial wastewater. Large median gaps between the models were observed for industrial wastewater, fecal sludge, and human excreta, associated with informal waste management, whereby LEJ did not work well. Overall, the sMFA demonstrated a good estimation of nitrogen and phosphorus flows with limited increase of uncertainty, still requiring focused attention on informal waste streams. Supplementary Information: The online version contains supplementary material available at 10.1007/s10163-023-01660-5.

Expert judgment in climate science: How it is used and how it can be justified.

Like any science marked by high uncertainty, climate science is characterized by a widespread use of expert judgment. In this paper, we first show that, in climate science, expert judgment is used to overcome uncertainty, thus playing a crucial role in the domain and even at times supplanting models. One is left to wonder to what extent it is legitimate to assign expert judgment such a status as an epistemic superiority in the climate context, especially as the production of expert judgment is particularly opaque. To begin answering this question, we highlight the key components of expert judgment. We then argue that the justification for the status and use of expert judgment depends on the competence and the individual subjective features of the expert producing the judgment since expert judgment involves not only the expert's theoretical knowledge and tacit knowledge, but also their intuition and values. This goes against the objective ideal in science and the criteria from social epistemology which largely attempt to remove subjectivity from expertise.

Validación de un instrumento para medir el respeto de la autonomía del paciente en situación terminal durante la toma de decisiones médicas sobre el final de la vida / Validació d'un instrument per mesurar el respecte de l'autonomia del pacient en situació terminal durant la presa de decisions mèdiques sobre el final de la vida / Validation of an instrument to measure respect for the autonomy of terminally ill patients during end-of-life medical decision-making

Antecedentes: El respeto por la autonomía de la persona consiste en considerarlas preferencias y valores de la persona enferma durante la toma de decisions sobre el tipo de atención que recibe y es un elemento bioético-jurídico. Sin embargo, no existen instrumentos validados sobre este fenómeno que ayuden aclarificar la percepción del médico sobre este principio.Objetivo: Elaborar, validar mediante juicio de expertos y pilotear para obtener la consistencia interna de un instrumento que evalúa el nivel de acuerdo de los médicos sobre los diferentes elementos que constituyen el respeto por la autonomía de la persona enferma en etapa terminal.Métodos: Estudio transversal. Método de validación por juicio de diez expertos de México. El instrumento se piloteó en médicos de un hospital público de alta especialidad para determinar la consistencia interna del mismo.Resultados: Se generó un instrumento de 15 ítems con un índice de validez de contenido de 0.82 para 10 expertos. Fue piloteado en una muestra de 96 médicos. Se obtuvo un alfa de Cronbach de 0.694.Conclusiones: Se desarrolló, validó y evaluó la consistencia interna de un cuestionario para medir el nivel de acuerdo de médicos que atienden a persones enfermas en etapa terminal con relación a los aspectos constitutivos sobre el respeto de su autonomía. Se recomienda su aplicación en médicos para corroborar su utilidad y favorecerun diagnóstico situacional sobre la situación estudiada.(AU)

Antecedents: El respecte per l'autonomia de la persona consisteix a considerar-les preferències i valors de la persona malalta durant la presa de decisions sobre el tipus d'atenció que rep i és un element bioeticojurídic. No obstant això, no hi ha instruments validats sobre aquest fenomen que ajudin a aclarir la percepció del metge sobreaquest principi.Objectiu: Elaborar, validar mitjançant un judici d'experts i pilotejar per obtenir la consistència interna d'un instrument que avalua el nivell d'acord dels metges sobre els diferents elements que constitueixen el respecte per l'autonomiade la persona malalta en l'etapa terminal.Mètodes: Estudi transversal. Mètode de validació per judici de deu experts de Mèxic. L'instrument es va pilotejar en metges d'un hospital públic d'alta especialitat per determinar-ne la consistència interna.Resultats: Es va generar un instrument de 15 ítems amb un índex de validesa de contingut de 0.82 per a 10 experts. Va ser pilotejat en una mostra de 96 metges. S'obtingué un alfa de Cronbach de 0.694.Conclusions: Es va desenvolupar, validar i avaluar la consistència interna d'un qüestionari per mesurar el nivell d'acord de metges que atenen persones malaltes en etapa terminal amb relació als aspectes constitutius sobre el respecte de la seva autonomia. Se'n recomana l'aplicació en metges per corroborar-ne la utilitat i afavorir un diagnòstic situacional sobre la situació estudiada.(AU)

Background: Respect for the autonomy of the person consists in considering the preferences and values of the sick person when making decisions about the type of care they receive and is a bioethical-legal element. However, there are no validated instruments on this phenomenon that help to clarify the physician's perception of this principle.Objective: To elaborate, validate through expert judgment and determine the internal consistency of an instrument that evaluates the level of agreement that the doctors have on the different elements that constitute respect for theautonomy of the terminally ill person.Methods: Cross-sectional study. The expert judgment validation method was used with a panel of ten experts from Mexico. The instrument was piloted in doctors of a highly specialized public hospital to determinate itsinternal consistency.Results: An instrument of 15 items was generated with a content validity index of 0.82 for 10 experts. It was piloted in a sample of 96 doctors (31.3% female and 68.7% male) and with an average age of 30 years. The Cronbach's alpha was 0.694.Conclusions: A questionnaire was developed to measure the level of agreement that the doctor who cares for terminally ill people has in relation to the constitutive aspects of respect for their autonomy. Based on the results of the validation andinternal consistency, its application in medical populations is recommended to corroborate its usefulness and favoura simple situational diagnosis of the situation studied.(AU)

The political polarization of COVID-19 treatments among physicians and laypeople in the United States.

In the United States, liberals and conservatives disagree about facts. To what extent does expertise attenuate these disagreements? To study this question, we compare the polarization of beliefs about COVID-19 treatments among laypeople and critical care physicians. We find that political ideology predicts both groups' beliefs about a range of COVID-19 treatments. These associations persist after controlling for a rich set of covariates, including local politics. We study two potential explanations: a) that partisans are exposed to different information and b) that they interpret the same information in different ways, finding evidence for both. Polarization is driven by preferences for partisan cable news but not by exposure to scientific research. Using a set of embedded experiments, we demonstrate that partisans perceive scientific evidence differently when it pertains to a politicized treatment (ivermectin), relative to when the treatment is not identified. These results highlight the extent to which political ideology is increasingly relevant for understanding beliefs, even among expert decision makers such as physicians.

Adaptation of the standardized plant analysis-risk human reliability analysis technique for the surgical setting: expert judgment approach.

Objectives. Application of human reliability analysis (HRA) techniques originally developed for industrial settings to the healthcare sector may be controversial in terms of reliability and methodological level. The aim of the present study was to adapt a standardized plant analysis risk-human reliability analysis (SPAR-H) technique for application in surgical settings through suggesting more context-specific definitions for performance shaping factors (PSFs), designing precise levels and elicitation of multipliers through a domain expert judgment approach. Methods. A ratio magnitude estimation approach was used for carrying out domain expert judgment for multiplier elicitation. Experts from four teaching hospitals participated in the present study. Intra-class correlation was used in order to examine the inter-rater reliability of the estimated multipliers for each level of diagnosis and action task type. Results. Available time, threat stress, task complexity, experience/training, procedures, working conditions, human-machine interface, fatigue and teamwork were the nine suggested PSFs for the adapted SPAR-H technique. Conclusion. Context-specific definitions of the PSFs can enhance the reliability of human error probability assessments. Eventually, it could be concluded that multiplier elicitation through domain expert judgment is an efficient approach for adaptation of the HRA techniques for application in specific contexts.

Validación de Actividades Profesionales Confiables (APROC) para la Certificación de Ginecología y Obstetricia / Validation of Entustable Professional Activities (EPAs) for Gynecology and Obstetrics Certification

Resumen Como parte del diseño de un instrumento para la evaluación de la certificación ante el Consejo Mexicano de Ginecología y Obstetricia en México se llevó a cabo el proceso de validación del constructo a evaluar. De acuerdo con la bibliografía consultada, la educación médica se basa en competencias, por lo que es coherente practicar una evaluación en el mismo tenor. Se decidió utilizar el concepto de Actividades Profesionales Confiables (APROC) porque son un puente entre la competencia y la actividad observable que puede llevar a cabo el sustentante. Se aplicó una metodología de cuatro fases que se inicia con una revisión bibliográfica con la que se definió una lista de APROC para validarla en el contexto mexicano. La fase dos consistió en aplicar la técnica de grupos focales que representarán las regiones médicas definidas por la FEMECOG, después de recabar y analizar la información de los grupos en la fase tres. La última fase consistió en una validación por parte de 10 expertos en Ginecología y Obstetricia para conseguir un nivel de acuerdo mayor al 80% en cada APROC. Como resultados de este proceso de validación de constructo a evaluar se obtuvo una lista de 15 APROC con alcances (definición de las delimitaciones de las APROC, dónde inician y dónde terminan) con un nivel de aceptación superior al 80%. Además, se corroboró la eficacia de la metodología cualitativa para realizar validaciones en donde las partes interesadas que podrían participar para definir áreas o temas no cuenten con disponibilidad debido a sus actividades laborales.

Abstract As part of the design of an instrument to evaluate the certification of the Mexican Council of Gynaecology and Obstetrics in Mexico, the validation process of the construct to be evaluated was carried out. According to the literature consulted, medical education is based on competencies, so it is coherent to carry out an evaluation in the same vein. It was decided to use Entrustable Professional Activities (EPA), as they provide a link between the competence and the observable activity that the supporter can perform. A four-phase methodology was carried out, starting with a bibliographic review that defined a list of EPAs in order to validate them for the Mexican context. The second phase consisted of using the technique of focus groups, representing the medical regions defined by FEMECOG in the country, once the information from the groups had been collected and analysed in the third phase, and the final phase consisted of validation by 10 experts in gynaecology and obstetrics, in order to achieve a level of agreement of more than 80% in each EPA. As a result of this validation process of the construct to be evaluated, a list of 15 EPAs with scopes (definition of the boundaries of the EPAs, where they start and how far they end) was obtained with a level of acceptance greater than 80%, in addition to confirming the effectiveness of the qualitative methodology to carry out validations when the interested parties who could participate in defining areas or topics are not available due to their work activities.

Validación de dominios y competencias por grupos focales y juicio de expertos / Validation of domains and competencies by focus groups and expert judgment

Resumen El Consejo Mexicano de Ginecología y Obstetricia certifica a los especialistas para ejercer su especialidad y brindar una atención de alta calidad a las pacientes. En la actualidad, el Consejo está rediseñando el examen de Certificación en Ginecología y Obstetricia orientado a la evaluación de competencias profesionales a partir de las actividades profesionales confiables que permitan identificar los dominios de la competencia médica en un especialista. La competencia es una variedad de habilidades ejercidas a través de múltiples dominios o aspectos del desempeño profesional; sus descriptores requieren que contengan habilidades relevantes, contexto y la etapa o nivel de evaluación. Por su parte, los dominios son un conjunto de competencias clave. Este informe incluye los resultados de la primera validación en México de dominios y competencias para la evaluación de los ginecoobstetras a partir de una metodología cualitativa que comprende una revisión de la bibliografía, grupos focales, análisis de trabajos colegiados y validación por juicio de expertos. Los resultados se presentan divididos por los ocho dominios en los que se trabajó, se mencionan los comentarios más relevantes a discutir por los cuatro grupos focales. Éstos y la validación por expertos permitió reunir comentarios valiosos, coherentes y funcionales para el sistema de evaluación que quiere llevar a cabo el Consejo. Este ejercicio permitirá el posterior desarrollo de la tabla de especificaciones, reactivos o nuevos instrumentos de evaluación coherentes con un sistema de dominios, competencias y actividades profesionales confiables.

Abstract The Mexican Council of Gynecology and Obstetrics certifies specialists to practice their specialty and provide high-quality healthcare to patients. Currently, the Council is redesigning the Gynecology and Obstetrics Certification exam, oriented to the evaluation of professional competencies based on reliable professional activities that allow the identification of domains of a specialist's medical competencies. Competency can be defined as a variety of skills across multiple domains or aspects of professional performance. Its descriptors require to contain relevant skills, context, and the stage or level of assessment. Domains can be described as a set of competencies that are considered essential. This report includes the results of the first validation done in Mexico. The validation of domains and competencies for the evaluation of physicians in the area of Obstetrics and Gynecology is based on a qualitative methodology that includes a literature review, focus groups, analysis of collegiate works, and validation through expert judgment. The obtained results are divided into eight domains that mention the most relevant observations that were discussed by the four focus groups. The focus groups and the validation through expert judgment made it possible to gather valuable, coherent, and functional feedback for the evaluation system that the Council wants to carry out. This method will allow the subsequent development of the table of specifications, items, or new evaluation instruments congruent with a system of domains, competencies, and reliable professional activities.

The critic's voice: On the role and function of criticism of classical music recordings.

In the Western classical tradition music criticism represents one of the most complex and influential forms of performance assessment and evaluation. However, in the age of peer opinion sharing and quick communication channels it is not clear what place music critics' judgments still hold in the classical music market. This article presents expert music critics' view on their role, function, and influence. It is based on semi-structured interviews with 14 native English- and German-speaking critics who had an average of 32 years professional activity in classical music review. We present the first visual model to summarize music critics' descriptions of their role and responsibilities, writing processes, and their influences (on the market and on artists). The model distinguishes six roles (hats): consumer adviser, teacher, judge, writer, stakeholder, and artist advocate. It identifies core principles governing critical writing for music as well as challenges that arise from balancing the above six responsibilities whilst remaining true to an implicit code of conduct. Finally, it highlights the factors that inform critics' writing in terms of the topics they discuss and the discursive tools they employ. We show that music critics self-identify as highly skilled mediators between artists, producers and consumers, and justify their roles as judge and teacher based on a wealth of experience as against the influx of pervasive amateur reviews. Our research approach also offers occupation-based insights into professional music review standards, including the challenges of maintaining objectivity and resisting commercial pressures. This article offers a new viewpoint on music critics' judgments and recommendations that helps to explain their expectations and reflections.

Validation of Content for an App for Caregivers of Stroke Patients through the Delphi Method.

The aim of this study was to reach a consensus among experts (using the Delphi technique) to validate the informative content that should be included in an App to be used by informal caregivers of stroke patients in order to improve their quality of life, facilitating access to the health care system and involving them in their own health. This technique was developed between June and December 2021. The group of experts was selected on the basis of previously established criteria, and the coefficient of variation (v) was used as a measure of consensus. In addition, the concordance index was calculated to determine the stability of the different rounds. In the first round, the preliminary content, previously elaborated by the research group, was evaluated as very appropriate for the objectives set (N-P < 1.07). In addition, averages of 4.5 out of five and a coefficient of variation of less than 0.5 were obtained, confirming the consensus. In the second round, suggestions were made by the experts on how to improve the content of the information, obtaining 100% agreement with the results obtained in the first round. The results obtained allow a positive evaluation of the use of the Delphi method for the elaboration of the information to be housed in an App.

Content Validity through Expert Judgment for the Depression Clinical Evaluation Test.

Background/Objective: The evaluation of depression requires valid and reliable measuring instruments, which collect a wide spectrum of symptoms that this disorder displays, in order to carry out an accurate and differential diagnosis. The objective of this work is the construction of the Depression Clinical Evaluation Test (DCET), where affective, somatic, cognitive, behavioral and interpersonal symptoms are considered and also analyze its content validity through an expert judgment. Method: Based on different diagnostic and manual classifications, a specification table for a depression test was established. In its evaluation, 16 experts in Psychological Assessment, Psychometry and/or Psychopathology participated. A total of 300 items were created. The experts had to assess the items according to the criteria of Content, Relevance, Clarity, Comprehension, Sensitivity, and Offensiveness. In addition, 50 adults, evaluated the compression of the items. Results: The degree of understanding for all the items was high and the expert judgment favoured the suppression of 104 items, thus obtaining a shorter measuring instrument with a total of 196 items for ease of application. Conclusions: The content validity of the test is adequate and fits the agreed definition of depression.

Antecedentes: La evaluación de la depresión requiere de instrumentos de medida válidos, fiables y que recojan el amplio espectro de síntomas que este trastorno conlleva, para poder llevar a cabo un diagnóstico certero y diferencial. El objetivo de este trabajo es la construcción del Test de Evaluación Clínica de la Depresión (TECD), que contempla síntomas afectivos, somáticos, cognitivos, conductuales e interpersonales, y analizar su validez de contenido a través de un juicio de expertos. Método: A partir de diferentes clasificaciones diagnósticas y manuales se estableció la tabla de especificación del test para este cuestionario de depresión. En la evaluación de este participaron 16 expertos en Evaluación Psicológica, Psicometría y/o Psicopatología. Se crearon 300 ítems en total, que los expertos tuvieron que valorar atendiendo a los criterios de Contenido, Relevancia, Claridad, Comprensión, Sensitividad y Ofensividad. Además, 50 adultos, valoraron la compresión de los items. Resultados: El grado de comprensión de todos los ítems fue elevado y el juicio de expertos supuso la supresión de 104 ítems, obteniendo así un instrumento de medida más breve, con 196 ítems en total, lo que facilitará su aplicación. Conclusiones: La validez de contenido del test es adecuada y se adapta a la definición de depresión establecida.

Content Validity through Expert Judgment for the Depression Clinical Evaluation Test / Validez de contenido mediante juicio de expertos para el Test de Evaluación Clínica de la Depresión

AbstractBackground/Objective: The evaluation of depression requires valid and reliable measuring instruments, which collect a wide spectrum of symptoms that this disorder displays, in order to carry out an accurate and differential diagnosis. The objective of this work is the construction of the Depression Clinical Evaluation Test (DCET), where affective, somatic, cognitive, behavioral and interpersonal symptoms are considered and also analyze its content validity through an expert judgment. Method: Based on different diagnostic and manual classifications, a specification table for a depression test was established. In its evaluation, 16 experts in psychological evaluation, psychometry and/or psychopathology participated. A total of 300 items were created. The experts had to assess the items according to the criteria of Content, Relevance, Clarity, Comprehension, Sensitivity and Offensiveness. In addition, 50 adults, evaluated the compression of the items. Results: The degree of understanding for all the items was high and the expert judgment favoured the suppression of 104 items, thus obtaining a shorter measuring instrument with a total of 196 items for ease of application. Conclusions: The content validity of the test is adequate and fits the agreed definition of depression.

Antecedentes La evaluación de la depresión requiere de instrumentos de medida válidos, fiables y que recojan el amplio espectro de síntomas que este trastorno conlleva, para poder llevar a cabo un diagnóstico certero y diferencial. El objetivo de este trabajo es la construcción del Test de Evaluación Clínica de la Depresión (TECD), que contempla síntomas afectivos, somáticos, cognitivos, conductuales e interpersonales, y analizar su validez de contenido a través de un juicio de expertos. Método A partir de diferentes clasificaciones diagnósticas y manuales se estableció la tabla de especificación del test para este cuestionario de depresión. En la evaluación de este participaron 16 expertos en Evaluación Psicológica, Psicometría y/o Psicopatología. Se crearon 300 ítems en total, que los expertos tuvieron que valorar atendiendo a los criterios de Contenido, Relevancia, Claridad, Comprensión, Sensitividad y Ofensividad. Además, 50 adultos, valoraron la compresión de los items. Resultados El grado de comprensión de todos los ítems fue elevado y el juicio de expertos supuso la supresión de 104 ítems, obteniendo así un instrumento de medida más breve, con 196 ítems en total, lo que facilitará su aplicación. Conclusiones La validez de contenido del test es adecuada y se adapta a la definición de depresión establecida.

A proposal for the assessment of replication of effects in single-case experimental designs.

In science in general and in the context of single-case experimental designs, replication of the effects of the intervention within and/or across participants or experiments is crucial for establishing causality and for assessing the generality of the intervention effect. Specific developments and proposals for assessing whether an effect has been replicated or not (or to what extent) are scarce, in the general context of behavioral sciences, and practically null in the single-case experimental designs context. We propose an extension of the modified Brinley plot for assessing how many of the effects replicate. To make this assessment possible, a definition of replication is suggested, on the basis of expert judgment, rather than on statistical criteria. The definition of replication and its graphical representation are justified, presenting their strengths and limitations, and illustrated with real data. A user-friendly software is made available for obtaining automatically the graphical representation.

What is a Good Calibration Question?

Weighted aggregation of expert judgments based on their performance on calibration questions may improve mathematically aggregated judgments relative to equal weights. However, obtaining validated, relevant calibration questions can be difficult. If so, should analysts settle for equal weights? Or should they use calibration questions that are easier to obtain but less relevant? In this article, we examine what happens to the out-of-sample performance of weighted aggregations of the classical model (CM) compared to equal weighted aggregations when the set of calibration questions includes many so-called "irrelevant" questions, those that might ordinarily be considered to be outside the domain of the questions of interest. We find that performance weighted aggregations outperform equal weights on the combined CM score, but not on statistical accuracy (i.e., calibration). Importantly, there was no appreciable difference in performance when weights were developed on relevant versus irrelevant questions. Experts were unable to adapt their knowledge across vastly different domains, and in-sample validation did not accurately predict out-of-sample performance on irrelevant questions. We suggest that if relevant calibration questions cannot be found, then analysts should use equal weights, and draw on alternative techniques to improve judgments. Our study also indicates limits to the predictive accuracy of performance weighted aggregation, and the degree to which expertise can be adapted across domains. We note limitations in our study and urge further research into the effect of question type on the reliability of performance weighted aggregations.

Balancing the Elicitation Burden and the Richness of Expert Input When Quantifying Discrete Bayesian Networks.

Structured expert judgment (SEJ) is a method for obtaining estimates of uncertain quantities from groups of experts in a structured way designed to minimize the pervasive cognitive frailties of unstructured approaches. When the number of quantities required is large, the burden on the groups of experts is heavy, and resource constraints may mean that eliciting all the quantities of interest is impossible. Partial elicitations can be complemented with imputation methods for the remaining, unelicited quantities. In the case where the quantities of interest are conditional probability distributions, the natural relationship between the quantities can be exploited to impute missing probabilities. Here we test the Bayesian intelligence interpolation method and its variations for Bayesian network conditional probability tables, called "InterBeta." We compare the various outputs of InterBeta on two cases where conditional probability tables were elicited from groups of experts. We show that interpolated values are in good agreement with experts' values and give guidance on how InterBeta could be used to good effect to reduce expert burden in SEJ exercises.