Methodological procedure based on quantitizing/liquefying: a case study to assess work climate in an emergency department.

Introduction: In the assessment of health organizations, results-based indicators are mainly used, with no consideration of internal work dynamics. This type of assessment forfeits much of the rich, useful information needed to make decisions on improving the organization. In order to address this, a rigorous procedure based on mixed methods is laid out here on gathering, analyzing, and interpreting data associated with the implementation process. Methods: A 55-year-old doctor was selected at random from among the staff who volunteered to be interviewed at the emergency department at a public hospital located in southern Spain for an interview. Qualitative data obtained from the in-depth interview (indirect observation) were progressively systematized (liquefied and quantitized) based on a theoretical framework until a code matrix was obtained, without losing or distorting any information. Afterwards, data quality was controlled using Cohen's kappa (κ) coefficient. A quantitative polar coordinate analysis was then carried out using the free software HOISAN (v. 1.6.3.3) to obtain robust results, vectorizing the relationships between codes and specifying whenever such relationships were statistically significant (and if they resulted in behavior activation or inhibition). Finally, a supplementary quantitative and qualitative assessment was carried out. Results and discussion: The proposed method was applied to the needs assessment of teams in order to evaluate that work climate in the hospital's emergency department Health Services of a hospital. Data quality control yielded an adequate result (κ = 0.82). Significant activation and inhibition of behaviors occurred, both prospectively and retrospectively. For instance, We seek to understand the needs of our clients and We readily adapt to new circumstances showed a significant activation (vector length = 3.43, p < 0.01) both prospectively (Zsum = 0.48) and retrospectively (Zsum = 3.4).An adequate method to obtain detailed information about group dynamics in a work environment is presented, based on an in-depth interview. Practical applications for implementations to improve the functioning of organizations are presented.

From randomized control trial to mixed methods: a practical framework for program evaluation based on methodological quality / Del ensayo controlado aleatorizado a los métodos mixtos: un marco práctico para la evaluación de programas basado en la calidad metodológica

The evidence used when making decisions about the design, implementation and evaluation in intervention programs should be methodologically sound. Depending on the context of the intervention, different methodologies may apply. Nonetheless, the intervention context is often unstable and, to adapt to changing circumstances, it sometimes becomes necessary to modify the original plan. The framework proposed herein draws on approaches that can be considered two extremes of a continuum (experimental/quasi-experimental designs and studies based on observational methodology). In unstable intervention context conditions, this enables decisions from a methodological quality approach regarding design, measurement, and analysis. Structural dimensions, i.e., units (participants, users), treatment (program activities), outcomes (results, including decisions about the instruments to use and data gathering), setting (implementation context) and time will be detailed as part of the practical framework. The present study aims to specify the degree of correspondence/complementarity between components in these structural dimensions of a program evaluation from a practical complementarity perspective based on methodological quality.(AU)

La evidencia utilizada al tomar decisiones sobre el diseño, implementación y evaluación en los programas de intervención debe ser metodológicamente sólida. Dependiendo del contexto de la intervención, se pueden aplicar diferentes metodologías. Sin embargo, el contexto de la intervención es a menudo inestable y, para adaptarse a las circunstancias cambiantes, en algunas ocasiones se hace necesario modificar el plan original. El marco propuesto en este documento se basa en enfoques que pueden considerarse dos extremos de un continuo (diseños experimentales / cuasiexperimentales y estudios basados ​​en metodología observacional). En condiciones de contexto de intervención inestable, esto permite tomar decisiones desde un enfoque de calidad metodológica en cuanto a diseño, medición y análisis. Las dimensiones estructurales, i.e., las unidades (participantes, usuarios), el tratamiento (actividades del programa), los resultados (incluidas las decisiones sobre los instrumentos a utilizar y la recopilación de datos), el entorno (contexto de implementación) y el tiempo se detallarán como parte del marco práctico. El presente estudio tiene como objetivo especificar el grado de correspondencia / complementariedad entre componentes en estas dimensiones estructurales de la evaluación de un programa desde una perspectiva de complementariedad práctica basada en la calidad metodológica.(AU)

Methodological quality checklist for studies based on observational methodology (mqcom) / Checklist de calidad metodológica para estudios basados en metodología observacional

BACKGROUND: No existing instrument addresses the minimum number of items that guarantee methodological quality in studies based on observational methodology. Consequently, professionals who are not experts in observational methodology do not have a basic framework to guide their practice in this type of study. This study developed a checklist to measure the minimum number of items for methodological quality that studies based on observational methodology should consider and provided evidence of their validity based on test content and intercoder reliability. METHOD: Fifty-four judges with at least 1 year of experience in observational methodology and research based on this methodology evaluated the items of the developed checklist in terms of relevance, usefulness, and feasibility. Items were selected if they obtained at least .5 in the Osterlind indexes of the three aspects evaluated. Two coders applied the selected items to a random selection of articles that used observational methodology to investigate soccer, and intercoder reliability was examined using Cohen's kappa (k) coefficients. RESULTS: The final checklist included 16 items grouped into 11 criteria/dimensions, with adequate reliability coefficients. CONCLUSIONS: This study developed a useful instrument for non-expert professionals to enhance the methodological quality of studies based on observational methodology

ANTECEDENTES: no existen instrumentos referidos a los ítems mínimos que garanticen la calidad metodológica en estudios basados en metodología observacional. En consecuencia, los profesionales no expertos en metodología observacional no disponen de una guía básica que oriente su práctica en este tipo de estudios. Se desarrolló una escala para medir los ítems mínimos de calidad metodológica que deben considerar los estudios basados en metodología observacional aportando evidencias de su validez basada en el contenido de la prueba y fiabilidad intercodificadores. MÉTODO: cincuenta y cuatro jueces con al menos un año de experiencia en metodología observacional y su aplicación evaluaron los ítems del checklist elaborado respecto a su relevancia, utilidad y viabilidad. Se seleccionaron aquellos ítems que obtuvieron al menos ,5 en los índices de Osterlind en los tres aspectos evaluados. Dos codificadores los aplicaron a una selección aleatoria de artículos que utilizaron metodología observacional en fútbol y se estudió la fiabilidad intercodificadores mediante coeficientes kappa (k) de Cohen. RESULTADOS: la escala resultante constó de 16 ítems agrupados en 11 criterios/dimensiones, con coeficientes de fiabilidad adecuados. CONCLUSIONES: se desarrolló un instrumento útil dirigido a profesionales no expertos para potenciar la calidad metodológica de estudios basados en metodología observacional

Methodological quality checklist for studies based on observational methodology (MQCOM).

BACKGROUND: No existing instrument addresses the minimum number of items that guarantee methodological quality in studies based on observational methodology. Consequently, professionals who are not experts in observational methodology do not have a basic framework to guide their practice in this type of study. This study developed a checklist to measure the minimum number of items for methodological quality that studies based on observational methodology should consider and provided evidence of their validity based on test content and intercoder reliability. METHOD: Fifty-four judges with at least 1 year of experience in observational methodology and research based on this methodology evaluated the items of the developed checklist in terms of relevance, usefulness, and feasibility. Items were selected if they obtained at least .5 in the Osterlind indexes of the three aspects evaluated. Two coders applied the selected items to a random selection of articles that used observational methodology to investigate soccer, and intercoder reliability was examined using Cohen's kappa (k) coefficients. RESULTS: The final checklist included 16 items grouped into 11 criteria/dimensions, with adequate reliability coefficients. CONCLUSIONS: This study developed a useful instrument for non-expert professionals to enhance the methodological quality of studies based on observational methodology.

Preliminary Checklist for Reporting Observational Studies in Sports Areas: Content Validity.

Observational studies are based on systematic observation, understood as an organized recording and quantification of behavior in its natural context. Applied to the specific area of sports, observational studies present advantages when comparing studies based on other designs, such as the flexibility for adapting to different contexts and the possibility of using non-standardized instruments as well as a high degree of development in specific software and data analysis. Although the importance and usefulness of sports-related observational studies have been widely shown, there is no checklist to report these studies. Consequently, authors do not have a guide to follow in order to include all of the important elements in an observational study in sports areas, and reviewers do not have a reference tool for assessing this type of work. To resolve these issues, this article aims to develop a checklist to measure the quality of sports-related observational studies based on a content validity study. The participants were 22 judges with at least 3 years of experience in observational studies, sports areas, and methodology. They evaluated a list of 60 items systematically selected and classified into 12 dimensions. They were asked to score four aspects of each item on 5-point Likert scales to measure the following dimensions: representativeness, relevance, utility, and feasibility. The judges also had an open-format section for comments. The Osterlind index was calculated for each item and for each of the four aspects. Items were considered appropriate when obtaining a score of at least 0.5 in the four assessed aspects. After considering these inclusion criteria and all of the open-format comments, the resultant checklist consisted of 54 items grouped into the same initial 12 dimensions. Finally, we highlight the strengths of this work. We also present its main limitation: the need to apply the resultant checklist to obtain data and, thus, increase quality indicators of its psychometric properties. For this reason, as relevant actions for further development, we encourage expert readers to use it and provide feedback; we plan to apply it to different sport areas.

Development of a Work Climate Scale in Emergency Health Services.

An adequate work climate fosters productivity in organizations and increases employee satisfaction. Workers in emergency health services (EHS) have an extremely high degree of responsibility and consequent stress. Therefore, it is essential to foster a good work climate in this context. Despite this, scales with a full study of their psychometric properties (i.e., validity evidence based on test content, internal structure and relations to other variables, and reliability) are not available to measure work climate in EHS specifically. For this reason, our objective was to develop a scale to measure the quality of work climates in EHS. We carried out three studies. In Study 1, we used a mixed-method approach to identify the latent conceptual structure of the construct work climate. Thus, we integrated the results found in (a) a previous study, where a content analysis of seven in-depth interviews obtained from EHS professionals in two hospitals in Gibraltar Countryside County was carried out; and (b) the factor analysis of the responses given by 113 EHS professionals from these same centers to 18 items that measured the work climate in health organizations. As a result, we obtained 56 items grouped into four factors (work satisfaction, productivity/achievement of aims, interpersonal relationships, and performance at work). In Study 2, we presented validity evidence based on test content through experts' judgment. Fourteen experts from the methodology and health fields evaluated the representativeness, utility, and feasibility of each of the 56 items with respect to their factor (theoretical dimension). Forty items met the inclusion criterion, which was to obtain an Osterlind index value greater than or equal to 0.5 in the three aspects assessed. In Study 3, 201 EHS professionals from the same centers completed the resulting 40-item scale. This new instrument produced validity evidence based on the internal structure in a second-order factor model with four components (RMSEA = 0.079, GFI = 0.97, AGFI = 0.97, CFI = 0.97; NFI = 0.95, and NNFI = 0.97); absence of Differential Item Functioning (DIF) in 80% of the items; reliability (α = 0.96); and validity evidence based on relations to other variables, specifically the test-criterion relationship (ρ = 0.680). Finally, we discuss further developments of the instrument and its possible implications for EHS workers.