Enhancing Content Validity Assessment With Item Response Theory Modeling / Mejorando la Evaluación de la Validez del Contenido a Través del Modelado de la Teoría de Respuesta al Ítem

Background: Ensuring the validity of assessments requires a thorough examination of the test content. Subject matter experts (SMEs) are commonly employed to evaluate the relevance, representativeness, and appropriateness of the items. This article proposes incorporating item response theory (IRT) into model assessments conducted by SMEs. Using IRT allows for the estimation of discrimination and threshold parameters for each SME, providing evidence of their performance in differentiating relevant from irrelevant items, thus facilitating the detection of suboptimal SME performance while improving item relevance scores. Method: Use of IRT was compared to traditional validity indices (content validity index and Aiken’s V) in the evaluation of conscientiousness items. The aim was to assess the SMEs’ accuracy in identifying whether items were designed to measure conscientiousness or not, and predicting their factor loadings. Results: The IRT-based scores effectively identified conscientiousness items (R2 = 0.57) and accurately predicted their factor loadings (R2 = 0.45). These scores demonstrated incremental validity, explaining 11% more variance than Aiken’s V and up to 17% more than the content validity index. Conclusions: Modeling SME assessments with IRT improves item alignment and provides better predictions of factor loadings, enabling improvement of the content validity of measurement instruments.(AU)

Antecedentes: Garantizar la validez de evaluaciones requiere un examen exhaustivo del contenido de una prueba. Es común emplear expertos en la materia (EM) para evaluar la relevancia, representatividad y adecuación de los ítems. Este artículo propone integrar la teoría de respuesta al ítem (TRI) en las evaluaciones hechas por EM. La TRI ofrece parámetros de discriminación y umbral de los EM, evidenciando su desempeño al diferenciar ítems relevantes/ irrelevantes, detectando desempeños subóptimos, mejorando también la estimación de la relevancia de los ítems. Método: Se comparó el uso de la TRI frente a índices tradicionales (índice de validez de contenido y V de Aiken) en ítems de responsabilidad. Se evaluó la precisión de los EM al discriminar si los ítems medían responsabilidad o no, y si sus evaluaciones permitían predecir los pesos factoriales de los ítems. Resultados: Las puntuaciones de TRI identificaron bien los ítems de responsabilidad (R2 = 0,57) y predijeron sus cargas factoriales (R2 = 0,45). Además, mostraron validez incremental, explicando entre 11% y 17% más de varianza que los índices tradicionales. Conclusiones: La TRI en las evaluaciones de los EM mejora la alineación de ítems y predice mejor los pesos factoriales, mejorando validez del contenido de los instrumentos.(AU)

Rating mean of expert judges and asymmetric confidence intervals in content validity: An SPSS syntax

La estimación de la validez de contenido, obtenida mediante el análisis racional de jueces expertos, habitualmente se hace con coeficientes que estandarizan entre 0.0 y 1.0 el juicio de los jueces. Sin embargo, esta estimación también puede expresarse en la métrica de las respuestas de los jueces, en la forma de la media de respuesta, y con intervalos de confianza asimétricos alrededor de esta media. El objetivo del presente manuscrito es implementar un procedimiento para estas estimaciones (media de respuesta e intervalos de confianza asimétricos) en un programa escrito en sintaxis SPSS. Se explica la racionalidad del procedimiento, y se desarrolla un ejemplo aplicado del cálculo. El programa es de distribución libre, solicitándolo a los autores.(AU)

The estimation of content validity, obtained by rational analysis of expert judges, is usually done with coefficients that standardize between 0.0 and 1.0 the judges' judgment. However, this estimate can also be ex-pressed in the metric of the judges' responses, in the form of the response mean, and with asymmetric confidence intervals around this mean. The aim of the present manuscript is to implement a procedure for these esti-mates (response mean and asymmetric confidence intervals) in a program written in SPSS syntax. The rationale of the procedure is explained, and an applied example of the calculation is developed. The program is freely dis-tributed upon request to the authors.

A Job Task Analysis for Technicians in the Offshore Wind Industry.

BACKGROUND: To establish whether an organization has a valid Physical Employment Standard (PES), it is important to determine those aspects of the job that are critical to operational success. OBJECTIVE: To determine the tasks of the Offshore Wind Industry (OWI) and whether the ability to undertake these tasks is adequately assessed. METHODS: The task analysis was completed through: observations; the research team undertaking tasks; reviewing operational manuals; and focus groups. In addition, a review of existing PES for the OWI was completed to determine whether standards matched with the results of the task analysis. RESULTS: Five critical tasks were identified: transfer from the vessel to the Transition Piece; ascent of the internal ladder; manoeuvre through hatches; torque and tensioning; and hauling a casualty up the tower. With the exception of aerobic capacity, the physical components required by Technicians are not assessed by the current medical standards, nor are these assessments standardized across companies. CONCLUSIONS: The Job Task Analysis undertaken can be used to inform decisions regarding the physical fitness requirements (selection), assessments and training of Technicians, with a view to ensuring that they are physically capable of undertaking the critical tasks without undue risk of injury to themselves or others.

Validity and reliability of physical employment standards.

In this paper the role of validity and reliability in the development of physical employment standards (PESs) and the consideration of these factors in determining the final pass/fail criteria for a PES and ultimately the legal defensibility of a PES is examined. Particular attention is paid to the use of subject-matter experts, the levels of evidence used in the establishment of the minimum acceptable pace/intensity for the completion of critical tasks, and the considerations needed in physical test selection.

Developing 'subject matter experts': an improvement methodology programme for acute postoperative pain with patients post major surgery.

AIMS AND OBJECTIVES: To determine the effectiveness of an improvement methodology initiative, directed at refining the quality of acute pain management of patients in the first 24 hours post major surgery using the Revised American Pain Society Patient Outcome Questionnaire, pre- and post development of a 'subject matter experts' acute pain programme. BACKGROUND: Accurately measuring effectiveness of acute pain management post major surgery is intertwined with measuring overall patient satisfaction. A critical element of quality evaluation is obtaining direct feedback from patients about the here-and-now pain experiences post major surgery. METHODS: A prospective cross-sectional, observational study was conducted in a large university hospital in Ireland. The questionnaire was completed with patients within 24 hours post major surgery, i.e., cardiothoracic, breast, gynaecological, gastrointestinal and urology surgery. The nurse participants were selected based on their commitment to play a key role in acute pain management. The study consisted of: a pre programme phase (n = 100 patients), an intervention phase - 'subject matter experts' acute pain programme (n = 24 nurses) and a postprogramme phase (n = 100 patients). RESULTS: Over a quarter of patients were in severe pain for long periods in the first 24 hours post major surgery. These findings were linked not only to ineffective analgesia from some pain drug therapies but also to contradictory messages from nurses. Over half of the patients' pre- and postintervention reported satisfaction with acute pain management, whereas the remainder were dissatisfied and some sought answers to their suboptimum pain status. The 'subject matter experts' had a noteworthy impact on the patients' pain beliefs. CONCLUSIONS: The findings revealed that a 'subject matter experts' acute pain programme can have a positive impact on pain management in the immediate phase post major surgery. RELEVANCE TO CLINICAL PRACTICE: The role making of 'subject matter experts' in acute pain is a tactical approach towards achieving optimum patient pain control in the immediate phase post major surgery.

Development of a Summative Examination with Subject Matter Expert Validation.

Objective. To describe the development, implementation and impact of a summative examination on student learning and programmatic curricular outcomes. Methods. The summative examination was developed using a systematic approach. Item reliability was evaluated using standard psychometric analyses. Content validity was assessed using necessity scoring as determined by subject matter experts. Results. Almost 700 items written by 37 faculty members were evaluated. Passing standards increased annually (45% in 2009 to 67% in 2014) as the result of targeting item difficulty and necessity scores. The percentage of items exhibiting discrimination above 0.1 increased to 100% over the four years. Necessity scores above 2.75 out of 4 increased from 65% to 100% of items over six years of examination administration. Conclusion. This examination successfully assessed student and curricular outcomes. Faculty member engagement observed in this process supports a culture of assessment. This type of examination could be beneficial to other programs.

The value of using test response data for content validity: An application of the bifactor-MIRT to a nursing knowledge test.

AIM: This paper aimed 1) to argue for the value of using test response data for content validation, and b) to demonstrate this practice using bifactor-multidimensional item response theory (bifactor-MIRT) for nurse education. METHOD: The Nursing Knowledge Test (NKT) response data by 1491 nurse students from China were used for demonstration. Based on the content structure assumed by subject-matter experts (SME), a bifactor-MIRT model was constructed and tested. This involved five steps: dimensionality assessment, local dependence detection, model specification, calibrating and unit weighting. RESULTS: Dimensionality assessment results confirmed the content structure assumed by SME. Through local dependence detection and calibrating (i.e., item parameter check), items suspected of contaminating content were detected and those producing substantive harm were removed or constrained. Finally, content contributions by items to the overall scale and to their subscales were obtained through unit weighting. CONCLUSION: Deficiencies residing in SME for content validation must raise attention. The study suggests the value of modeling test response data to compensate these deficiencies. The theoretical implication is discussed.

Inter-rater reliability of query/probe-based techniques for measuring situation awareness.

Query- or probe-based situation awareness (SA) measures sometimes rely on process experts to evaluate operator actions and system states when used in representative settings. This introduces variability of human judgement into the measurements that require inter-rater reliability assessment. However, the literature neglects inter-rater reliability of query/probe-based SA measures. We recruited process experts to provide reference keys to SA queries in trials of a full-scope nuclear power plant simulator experiment to investigate the inter-rater reliability of a query-based SA measure. The query-based SA measure demonstrated only 'moderate' inter-rater reliability even though the queries were seemingly direct. The level of agreement was significantly different across pairs of experts who had different levels of exposure to the experiment. The results caution that inter-rater reliability of query/probe-based techniques for measuring SA cannot be assumed in representative settings. Knowledge about the experiment as well as the domain is critical to forming reliable expert judgements. PRACTITIONER SUMMARY: When the responses of domain experts are treated as the correct answers to the queries or probes of SA measures used in representative or industrial settings, practitioners should take caution in assuming (or otherwise assess) inter-rater reliability of the situation awareness measures.

Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills.

BACKGROUND: Technical skills training in neurosurgery is mostly done in the operating room. New educational paradigms are encouraging the development of novel training methods for surgical skills. Simulation could answer some of these needs. This article presents the development of a conceptual training framework for use on a virtual reality neurosurgical simulator. METHODS: Appropriate tasks were identified by reviewing neurosurgical oncology curricula requirements and performing cognitive task analyses of basic techniques and representative surgeries. The tasks were then elaborated into training modules by including learning objectives, instructions, levels of difficulty, and performance metrics. Surveys and interviews were iteratively conducted with subject matter experts to delimitate, review, discuss, and approve each of the development stages. RESULTS: Five tasks were selected as representative of basic and advanced neurosurgical skill. These tasks were: 1) ventriculostomy, 2) endoscopic nasal navigation, 3) tumor debulking, 4) hemostasis, and 5) microdissection. The complete training modules were structured into easy, intermediate, and advanced settings. Performance metrics were also integrated to provide feedback on outcome, efficiency, and errors. The subject matter experts deemed the proposed modules as pertinent and useful for neurosurgical skills training. CONCLUSIONS: The conceptual framework presented here, the Fundamentals of Neurosurgery, represents a first attempt to develop standardized training modules for technical skills acquisition in neurosurgical oncology. The National Research Council Canada is currently developing NeuroTouch, a virtual reality simulator for cranial microneurosurgery. The simulator presently includes the five Fundamentals of Neurosurgery modules at varying stages of completion. A first pilot study has shown that neurosurgical residents obtained higher performance scores on the simulator than medical students. Further work will validate its components and use in a training curriculum.