The impact of receiving the same items on consecutive computer adaptive test administrations.

This study addresses item exposure in a Computerized Adaptive Test (CAT) when the item selection algorithm is permitted to present examinees with questions that they have already been asked in a previous test administration. The results indicate that the combined use of an adaptive algorithm to select items and latent trait theory to estimate person ability provides substantial protection from score contamination. The implications for constraints that prohibit examinees from seeing an item twice are discussed.

The Job Responsibilities Scale: invariance in a longitudinal prospective study.

The purpose of the present analysis was to determine the degree of invariance of the Job Responsibilities Scale from 1993 to 1995. Factor analyses were conducted on both year's data and nearly identical factor patterns were found. Rasch rating scale analyses were conducted and nearly identical pairs of item estimates were found. These results suggest that even though the overall frequency of performance on some medical technology laboratory tasks increased from 1993 to 1995, the relationships among the tasks themselves remained the same (invariant). This conclusion allows for a description of what it means to increase in level of personal job responsibility from year-to-year. In addition, these results suggest that at the conclusion of this prospective study it may be possible to objectively define the typical career mobility pattern of entry level medical technologists.

Postgraduate education: does it improve the knowledge base of practitioners with time?

PURPOSE: We evaluate the effect of the provision of postgraduate educational material on improving practitioner knowledge base during a 3-year period. MATERIALS AND METHODS: A total of 210 urologists were provided 67 monographs in a 2-year period. They were given a pretest before and posttest 1 year after receipt of all monographs. RESULTS: There was significant improvement in posttest scores for the group as a whole. Improvement correlated with years post-training and number of monographs read. Multivariate analysis revealed the number of monographs read was the only independent predictor of outcome. CONCLUSIONS: Although the improvement in test scores was significant and did correlate with the postgraduate educational material read, it was modest. This finding raises significant concerns about the usefulness of this format for graduate medical education.

Interpreting the chi-square statistics reported in the many-faceted Rasch model.

The different chi-square statistics reported in the many-faceted Rasch model analysis are presented and interpreted. In addition, other chi-square summary values are computed and presented for interpretation of facets. The chi-square values are useful for determining: (1) the significance of a facet in the Rasch model; (2) the significant contribution of facet main and interaction effects; (3) differences among facet elements; and (4) identifying the specific facet interaction adjustments to the subjects' calibrated logit ability measure.

Scoring and analysis of performance examinations: a comparison of methods and interpretations.

The purpose of this study was to compare the results and interpretation of the data from a performance examination when four methods of analysis are used. Methods are 1) traditional summary statistics, 2) inter-judge correlations, 3) generalizability theory, and 4) the multi-facet Rasch model. Results indicated that similar sources of variance were identified using each method; however, the multi-facet Rasch model is the only method that linearized the scores and accounts for differences in the particular examination challenged by a candidate before ability estimates are calculated.

Medical technologist career commitment and satisfaction with job benefits.

The purpose of this study was to determine the impact of career commitment on satisfaction with job-related benefits. A validated seven-point career commitment scale was used to measure career commitment. A 31-item benefits survey was used to collect data on satisfaction. In addition, salary information was collected. An analysis of variance was used to analyze the data. The high career commitment group was significantly more satisfied with 29 of the 31 benefits. There was no significant difference in salary among high, medium, and low career commitment groups. Low commitment individuals comprised only 11.5% of the respondents. The positive attitudes of the majority of medical technologists are a useful asset to the laboratory during this time of change in the practice of laboratory medicine.

The effect of rater severity on person ability measure: a Rasch model analysis.

This paper presents a method for analyzing oral examinations with an extended, many-faceted Rasch model that calibrates medical specialty candidates, protocols, and raters. Significant variance was found among protocol difficulties and rater severities. When candidates' raw scores were compared with calibrated measures corrected for the bias caused by the particular protocols and raters encountered, variation between candidate scores and measures were observed. The data were found to fit the Rasch model well enough to be suitable for making measurement on oral examinations more objective as well as providing specific feedback to oral examination raters. In this example a medical oral examination was used; however, the techniques are applicable to any situation in which trained professionals rate candidate or patient performances. For occupational therapists, potential applications include evaluation of a student's fieldwork performance or observation of a patient's task performance.

Laboratory staff qualifications and accuracy of proficiency test results. A national study.

In the last quarter of 1988, laboratories from across the country that participated in the College of American Pathologists (Northbrook, Ill) proficiency testing program were asked to complete a questionnaire describing their staffing pattern and submit it with their test results. Laboratories were then divided into two groups, those employing 100% American Society of Clinical Pathologists (Chicago, Ill) certified technologists and those employing 0% American Society of Clinical Pathologists certified technologists. The nonparametric Wilcoxon significance test was used to determine differences in accuracy of test results for laboratories in each group. Significant differences were found for the basic laboratory survey, comprehensive diagnostic immunology, comprehensive blood banking, comprehensive hematology, and comprehensive chemistry. chi 2 Analysis revealed that laboratories employing 100% American Society of Clinical Pathologists certified technologists had a significantly higher probability of achieving 80% accuracy across all surveys. These results present a strong argument that staffing with qualified technologists contributes to maximizing the quality of laboratory services offered to the public.

Comparability of decisions for computer adaptive and written examinations.

The purpose of this study was to determine the comparability of examinee ability measures, pass/fail decisions, and confidence in the accuracy of pass/fail decisions on written fixed-length and computer adaptive examinations. Medical technology students took two independent tests: a written test of 109 questions and a computer-administered computer adaptive test that included 50 to 100 items tailored to the ability of each student. Results indicated that ability measures on the two tests correlated at 0.84. Decisions were made with 90% confidence in their accuracy for 72% of the examinees on the computer adaptive test, and for 58% of the examinees on the traditional written test. The decision accuracy on the computer adaptive test was higher even though the test was shorter because the tailoring process reduced the error of measure.

A comparison of intra- and interjudge decision consistencies using analytic and holistic scoring criteria.

This study explored the effect of scoring criteria on the decision consistency of judges. Intra- and interjudge consistencies were compared when two different scoring criteria were used by the same judges. Analytic criteria included detailed assessments of a student's performance, while holistic criteria included only a subjective global assessment. Data from 18 judges and 9 histotechnology practical examinations were analyzed. Lack of intrajudge decision consistency was observed when the same practical examinations were assessed using analytic and holistic criteria. Interjudge decision consistency also varied when the different scoring criteria were used. More practical examinations were judged as passing when analytic scoring criteria were used. This particular analysis did not support a definitive conclusion concerning the value of holistic and analytic criteria.

The first six years' experience with the Pathology Residents' In-Service Examination, 1983-1988.

Data derived from the 1983-1988 administrations of the American Society of Clinical Pathology Residents In-Service Examination (PRISE) are presented. Data indicate an increase in participation by U.S. and Canadian pathology residency training programs (from 55% in 1983 to approximately 90% in 1988). Demographic data reflect an increase in the proportion of foreign medical graduates (from 24.8% in 1983 to 32.8% in 1988) in U.S. training programs with concomitant decrease in proportion of U.S. medical graduates (from 74.8% to 61.5%, respectively). Canadian residents comprise approximately 5% of the test population. Response surveys from residents and program directors indicate a high degree of satisfaction with the PRISE. In 1988 40.6% of all program directors thought that the PRISE satisfied their objectives very well, and 56% thought that objectives were met satisfactorily. For residents the corresponding rates were 21.2% and 70.7%. Trends in resident group performance in each subspecialty area of pathology, and the implications for residency training are discussed.

The impact of the quality of laboratory staff on the accuracy of laboratory results.

This study tests the premise that laboratories employing medical technologists certified by the Board of Registry of the American Society of Clinical Pathologists (MT[ASCP]) produce more accurate laboratory test results, as measured by the College of American Pathologists proficiency tests. Licensed laboratories in Illinois provided the sample. An accuracy score on the College of American Pathologists proficiency tests was calculated for each laboratory. The accuracy score of a subgroup of laboratories employing all (100%) certified medical technologists was compared with the accuracy score of a subgroup of laboratories employing only noncertified medical technologists. Those laboratories employing only certified medical technologists had a mean accuracy score of 95% (SD = 4%), while laboratories employing only noncertified medical technologists had a mean accuracy score of 75% (SD = 30%). The Mann-Whitney U test was used to identify differences between the two groups of laboratories. A difference in the accuracy scores between the two groups of laboratories was statistically discernible. Since most laboratories employ some certified medical technologists, a second analysis considered the relationship of the proportion of certified medical technologists employed in the laboratory and accuracy on College of American Pathologists proficiency tests. A significant positive Spearman rs correlation confirmed a relationship between employing a higher proportion of certified medical technologists and accuracy of test results.