Scoping Meta-Review of Methods Used to Assess Artificial Intelligence-Based Medical Devices for Heart Failure.

Artificial intelligence and machine learning (AI/ML) are playing increasingly important roles, permeating the field of medical devices (MDs). This rapid progress has not yet been matched by the Health Technology Assessment (HTA) process, which still needs to define a common methodology for assessing AI/ML-based MDs. To collect existing evidence from the literature about the methods used to assess AI-based MDs, with a specific focus on those used for the management of heart failure (HF), the International Federation of Medical and Biological Engineering (IFMBE) conducted a scoping meta-review. This manuscript presents the results of this search, which covered the period from January 1974 to October 2022. After careful independent screening, 21 reviews, mainly conducted in North America and Europe, were retained and included. Among the findings were that deep learning is the most commonly utilised method and that electronic health records and registries are among the most prevalent sources of data for AI/ML algorithms. Out of the 21 included reviews, 19 focused on risk prediction and/or the early diagnosis of HF. Furthermore, 10 reviews provided evidence of the impact on the incidence/progression of HF, and 13 on the length of stay. From an HTA perspective, the main areas requiring improvement are the quality assessment of studies on AI/ML (included in 11 out of 21 reviews) and their data sources, as well as the definition of the criteria used to assess the selection of the most appropriate AI/ML algorithm.

Approaches Adopted by Researchers to Measure the Quality of the Experience of People Working from Home: a Scoping Review.

Working from home (WFH) remotely is a modality of working that requires the careful design of systems of rules and tools to enable people to exchange information and perform actions. WFH is expected to expand after the COVID-19 pandemic. How to assess and compare in a reliable way the experience of workers with different (sociotechnical) systems of WFH is a central point to supporting the diffusion of acceptable modalities of working. However, the concept of experience and how it can be measured in the domain in WFH is yet to be clearly characterized. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology for scoping reviews, we systematically map the approaches used by researchers to assess WFH, identify which aspects are usually investigated, and examine how such aspects are usually measured in terms of questions and tools. Literature is collected using Scopus and Web of Science. Thirty-four records out of 323 focusing either on validating a scale, presenting theoretically the experience of workers or testing this empirically are included in the qualitative synthesis. The results highlight a lack of unified terminology and tools, with assessments of workers' experience mainly characterized by survey approaches and qualitative questions. Clustering together the most investigated aspects in the literature and reviewing how these aspects are assessed, we propose a list of 10 relevant overarching dimensions and attempt to define workers' experience in the domain of WFH remotely. This definition can be used as a tool by researchers aiming to assess the experience of workers in order to inform the design or redesign of the sociotechnical systems that enable WFH. Supplementary Information: The online version contains supplementary material available at 10.1007/s41347-022-00264-4.

Development and validation of ester impregnated pH strips for locating nasogastric feeding tubes in the stomach-a multicentre prospective diagnostic performance study.

BACKGROUND: NG (nasogastric) tubes are used worldwide as a means to provide enteral nutrition. Testing the pH of tube aspirates prior to feeding is commonly used to verify tube location before feeding or medication. A pH at or lower than 5.5 was taken as evidence for stomach intubation. However, the existing standard pH strips lack sensitivity, especially in patients receiving feeding and antacids medication. We developed and validated a first-generation ester-impregnated pH strip test to improve the accuracy towards gastric placements in adult population receiving routine NG-tube feeding. The sensitivity was improved by its augmentation with the action of human gastric lipase (HGL), an enzyme specific to the stomach. METHODS: We carried out a multi-centred, prospective, two-gate diagnostic accuracy study on patients who require routine NG-tube feeding in 10 NHS hospitals comparing the sensitivity of the novel pH strip to the standard pH test, using either chest X-rays or, in its absence, clinical observation of the absence of adverse events as the reference standard. We also tested the novel pH strips in lung aspirates from patients undergoing oesophageal cancer surgeries using visual inspection as the reference standard. We simulated health economics using a decision analytic model and carried out adoption studies to understand its route to commercialisation. The primary end point is the sensitivity of novel and standard pH tests at the recommended pH cut-off of 5.5. RESULTS: A total of 6400 ester-impregnated pH strips were prepared based on an ISO13485 quality management system. A total of 376 gastric samples were collected from adult patients in 10 NHS hospitals who were receiving routine NG-tube feeding. The sensitivities of the standard and novel pH tests were respectively 49.2% (95% CI 44.1­54.3%) and 70.2% (95% CI 65.6­74.8%) under pH cut-off of 5.5 and the novel test has a lung specificity of 89.5% (95% CI 79.6%, 99.4%). Our simulation showed that using the novel test can potentially save 132 unnecessary chest X-rays per check per every 1000 eligible patients, or direct savings of £4034 to the NHS. CONCLUSIONS: The novel pH test correctly identified significantly more patients with tubes located inside the stomach compared to the standard pH test used widely by the NHS. TRIAL REGISTRATION: http://www.isrctn.com/ISRCTN11170249 , Registered 21 June 2017-retrospectively registered.

Attitudes towards Trusting Artificial Intelligence Insights and Factors to Prevent the Passive Adherence of GPs: A Pilot Study.

Artificial Intelligence (AI) systems could improve system efficiency by supporting clinicians in making appropriate referrals. However, they are imperfect by nature and misdiagnoses, if not correctly identified, can have consequences for patient care. In this paper, findings from an online survey are presented to understand the aptitude of GPs (n = 50) in appropriately trusting or not trusting the output of a fictitious AI-based decision support tool when assessing skin lesions, and to identify which individual characteristics could make GPs less prone to adhere to erroneous diagnostics results. The findings suggest that, when the AI was correct, the GPs' ability to correctly diagnose a skin lesion significantly improved after receiving correct AI information, from 73.6% to 86.8% (X2 (1, N = 50) = 21.787, p < 0.001), with significant effects for both the benign (X2 (1, N = 50) = 21, p < 0.001) and malignant cases (X2 (1, N = 50) = 4.654, p = 0.031). However, when the AI provided erroneous information, only 10% of the GPs were able to correctly disagree with the indication of the AI in terms of diagnosis (d-AIW M: 0.12, SD: 0.37), and only 14% of participants were able to correctly decide the management plan despite the AI insights (d-AIW M:0.12, SD: 0.32). The analysis of the difference between groups in terms of individual characteristics suggested that GPs with domain knowledge in dermatology were better at rejecting the wrong insights from AI.

Inside pandora's box: a systematic review of the assessment of the perceived quality of chatbots for people with disabilities or special needs.

Introduction: People with disabilities or special needs can benefit from AI-based conversational agents (i.e., chatbots) that are used for competence training and well-being management. Assessing the quality of interactions with these chatbots is key to being able to reduce dissatisfaction with them and to understanding their potential long-term benefit. This in turn will help to increase adherence to their use, thereby improving the quality of life of the large population of end-users that they are able to serve.Methods: Following Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) methodology, we systematically reviewed the literature on methods of assessing the perceived quality of interactions with chatbots using the from Scopus and the Web of Science electronic databases. Using the Boolean operators (AND/OR) the keywords chatbot*, conversational agent*, special needs, disability were combined.Results: Revealed that only 15 of 192 papers on this topic included people with disabilities or special needs in their assessments. The results also highlighted the lack of a shared theoretical framework for assessing the perceived quality of interactions with chatbots.Conclusion: Systematic procedures based on reliable and valid methodologies continue to be needed in this field. The current lack of reliable tools and systematic methods to assess chatbots for people with disabilities and special needs is concerning, and ultimately, it may also lead to unreliable systems entering the market with disruptive consequences for people.Implications for rehabilitationChatbots applied in rehabilitation are mainly tested in terms of clinical effectiveness and validity with a minimal focus on measuring the quality of the interactionThe usability and interactive properties of chatbots applied in rehabilitation are not comparable as each tool is measured in different wayThe lack of a common framework to assess chatbots exposes people with disability and special needs to the risk of using unreliable tools.

Tablet and web-based audiometry to screen for hearing loss in adults with cystic fibrosis.

INTRODUCTION: Individuals with chronic lung disease (eg, cystic fibrosis (CF)) often receive antimicrobial therapy including aminoglycosides resulting in ototoxicity. Extended high-frequency audiometry has increased sensitivity for ototoxicity detection, but diagnostic audiometry in a sound-booth is costly, time-consuming and requires a trained audiologist. This cross-sectional study analysed tablet-based audiometry (Shoebox MD) performed by non-audiologists in an outpatient setting, alongside home web-based audiometry (3D Tune-In) to screen for hearing loss in adults with CF. METHODS: Hearing was analysed in 126 CF adults using validated questionnaires, a web self-hearing test (0.5 to 4 kHz), tablet (0.25 to 12 kHz) and sound-booth audiometry (0.25 to 12 kHz). A threshold of ≥25 dB hearing loss at ≥1 audiometric frequency was considered abnormal. Demographics and mitochondrial DNA sequencing were used to analyse risk factors, and accuracy and usability of hearing tests determined. RESULTS: Prevalence of hearing loss within any frequency band tested was 48%. Multivariate analysis showed age (OR 1.127; (95% CI: 1.07 to 1.18; p value<0.0001) per year older) and total intravenous antibiotic days over 10 years (OR 1.006; (95% CI: 1.002 to 1.010; p value=0.004) per further intravenous day) were significantly associated with increased risk of hearing loss. Tablet audiometry had good usability, was 93% sensitive, 88% specific with 94% negative predictive value to screen for hearing loss compared with web self-test audiometry and questionnaires which had poor sensitivity (17% and 13%, respectively). Intraclass correlation (ICC) of tablet versus sound-booth audiometry showed high correlation (ICC >0.9) at all frequencies ≥4 kHz. CONCLUSIONS: Adults with CF have a high prevalence of drug-related hearing loss and tablet-based audiometry can be a practical, accurate screening tool within integrated ototoxicity monitoring programmes for early detection.

The Lean and Agile Multi-dimensional Process (LAMP) - a new framework for rapid and iterative evidence generation to support health-care technology design and development.

Introduction: Health technology assessments (HTA) are tools for policymaking and resource allocation. Early HTAs are increasingly used in design and development of new technologies. Conducting early HTAs is challenging, due to a lack of evidence and significant uncertainties in the technology and the market. A multi-disciplinary approach is considered essential. However, an operational framework that can enable the integration of multi-dimensional evidence into commercialization remains lacking.Areas covered: We developed the Lean and Agile Multi-dimensional Process (LAMP), an early HTA framework, for embedding commercial decision-making in structured evidence generation activities, divided into phases. Diverse evidence in unmet needs, user acceptance, cost-effectiveness, and market competitiveness are being generated in increasing depth. This supports the emergence of design and value propositions that align technology capabilities and clinical and user needs.Expert opinion: We have been applying LAMP to working with medical device and diagnostic industry in the UK. The framework can be adapted to suit different technologies, decision needs, time scales, and resources. LAMP offers a practical solution to the multi-disciplinary approach. Methodologists drive the process by performing evidence generation and synthesis as and by enabling interactions between manufacturers, designers, clinicians, and other key stakeholders.

Is the LITE version of the usability metric for user experience (UMUX-LITE) a reliable tool to support rapid assessment of new healthcare technology?

OBJECTIVE: To ascertain the reliability of a standardised, short-scale measure of satisfaction in the use of new healthcare technology i.e., the LITE version of the usability metric for user experience (UMUX-LITE). Whilst previous studies have demonstrated the reliability of UMUX-LITE, and its relationship with measures of likelihood to recommend a product, such as the Net Promoter Score (NPS) in other sectors no such testing has been undertaken with healthcare technology. MATERIALS AND METHODS: Six point-of-care products at different stages of development were assessed by 120 healthcare professionals. UMUX-LITE was used to gather their satisfaction in use, and NPS to declare their intention to promote the product. Inferential statistics were used to: i) ascertain the reliability of UMUX-LITE, and ii) assess the relationship between UMUX-LITE and NPS at different stages of products development. RESULTS: UMUX-LITE showed an acceptable reliability (α = 0.7) and a strong positive correlation with NPS (r = 0.455, p < .001). This is similar to findings in other fields of application. The level of product development did not affect the UMUX-LITE scores, while the stage of development was a significant predictor (R2 = 0.49) of the intention to promote. DISCUSSION AND CONCLUSION: Practitioners may apply UMUX-LITE alone, or in combination with the NPS, to complement interview and 'homemade' scales to investigate the quality of new products at different stages of development. This shortened scale is appropriate for use in the context of healthcare in which busy professionals have a minimal amount of time to support innovation.

HEALTH TECHNOLOGY ASSESSMENT METHODS GUIDELINES FOR MEDICAL DEVICES: HOW CAN WE ADDRESS THE GAPS? THE INTERNATIONAL FEDERATION OF MEDICAL AND BIOLOGICAL ENGINEERING PERSPECTIVE.

OBJECTIVES: Current health technology assessment (HTA) methods guidelines for medical devices may benefit from contributions by biomedical and clinical engineers. Our study aims to: (i) review and identify gaps in the current HTA guidelines on medical devices, (ii) propose recommendations to optimize the impact of HTA for medical devices, and (iii) reach a consensus among biomedical engineers on these recommendations. METHODS: A gray literature search of HTA agency Web sites for assessment methods guidelines on devices was conducted. The International Federation of Medical and Biological Engineers (IFMBE) then convened a structured focus group, with experts from different fields, to identify potential gaps in the current HTA guidelines, and to develop recommendations to fill these perceived gaps. The thirty recommendations generated from the focus group were circulated in a Delphi survey to eighty-five biomedical and clinical engineers. RESULTS: Thirty-two panelists, from seventeen countries, participated in the Delphi survey. The responses showed a strong agreement on twenty-seven of thirty recommendations. Some uncertainties remain about the methods to accurately assess the effectiveness and safety, and interoperability of a medical device with other devices or within the clinical setting. CONCLUSIONS: As medical devices differ from drug therapies, current HTA methods may not accurately reflect the conclusions of their assessment. Recommendations informed by the focus group discussions and Delphi survey responses aimed to address the perceived gaps, and to provide a more integrated approach in medical device assessments in combining engineering with other perspectives, such as clinical, economic, patient, human factors, ethical, and environmental.

Evaluation of a hub-and-spoke model for the delivery of femtosecond laser-assisted cataract surgery within the context of a large randomised controlled trial.

AIMS: To test a hypothesis that cataract operating room (OR) productivity can be improved with a femtosecond laser (FL) using a hub-and-spoke model and whether any increase in productivity can offset additional costs relating to the FL. METHODS: 400 eyes of 400 patients were enrolled in a randomised-controlled trial comparing FL-assisted cataract surgery (FLACS) with conventional phacoemulsification surgery (CPS). 299 of 400 operations were performed on designated high-volume theatre lists (FLACS=134, CPS=165), where a hub-and-spoke FLACS model (1×FL, 2×ORs=2:1) was compared with independent CPS theatre lists. Details of operative timings and OR utilisation were recorded. Differences in productivity between hub-and-spoke FLACS and CPS sessions were compared using an economic model including testing hypothetical 3:1 and 4:1 models. RESULTS: The duration of the operation itself was 12.04±4.89 min for FLACS compared with CPS of 14.54±6.1 min (P<0.001). Total patient time in the OR was reduced from 23.39±6.89 min with CPS to 20.34±5.82 min with FLACS (P<0.001)(reduction of 3.05 min per case). There was no difference in OR turnaround time between the models. Average number of patients treated per theatre list was 9 for FLACS and 8 for CPS. OR utilisation was 92.08% for FLACS and 95.83% for CPS (P<0.001). Using a previously established economic model, the FLACS service cost £144.60 more than CPS per case. This difference would be £131 and £125 for 3:1 and 4:1 models, respectively. CONCLUSION: The FLACS hub-and-spoke model was significantly faster than CPS, with patients spending less time in the OR. This enabled an improvement in productivity, but insufficient to meaningfully offset the additional costs relating to FLACS.

Time and motion studies of National Health Service cataract theatre lists to determine strategies to improve efficiency.

AIM: To provide a quantitative assessment of cataract theatre lists focusing on productivity and staffing levels/tasks using time and motion studies. METHODS: National Health Service (NHS) cataract theatre lists were prospectively observed in five different institutions (four NHS hospitals and one private hospital). Individual tasks and their timings of every member of staff were recorded. Multiple linear regression analyses were performed to investigate possible associations between individual timings and tasks. RESULTS: 140 operations were studied over 18 theatre sessions. The median number of scheduled cataract operations was 7 (range: 5-14). The average duration of an operation was 10.3 min±(SD 4.11 min). The average time to complete one case including patient turnaround was 19.97 min (SD 8.77 min). The proportion of the surgeons' time occupied on total duties or operating ranged from 65.2% to 76.1% and from 42.4% to 56.7%, respectively. The correlations of the surgical time to patient time in theatre was R2=0.95. A multiple linear regression model found a significant association (F(3,111)=32.86, P<0.001) with R2=0.47 between the duration of one operation and the number of allied healthcare professionals (AHPs), the number of AHP key tasks and the time taken to perform these key tasks by the AHPs. CONCLUSIONS: Significant variability in the number of cases performed and the efficiency of patient flow were found between different institutions. Time and motion studies identified requirements for high-volume models and factors relating to performance. Supporting the surgeon with sufficient AHPs and tasks performed by AHPs could improve surgical efficiency up to approximately double productivity over conventional theatre models.

Designing medical technology for resilience: integrating health economics and human factors approaches.

INTRODUCTION: The slow adoption of innovation into healthcare calls into question the manner of evidence generation for medical technology. This paper identifies potential reasons for this including a lack of attention to human factors, poor evaluation of economic benefits, lack of understanding of the existing healthcare system and a failure to recognise the need to generate resilient products. Areas covered: Recognising a cross-disciplinary need to enhance evidence generation early in a technology's life cycle, the present paper proposes a new approach that integrates human factors and health economic evaluation as part of a wider systems approach to the design of technology. This approach (Human and Economic Resilience Design for Medical Technology or HERD MedTech) supports early stages of product development and is based on the recent experiences of the National Institute for Health Research London Diagnostic Evidence Co-operative in the UK. Expert commentary: HERD MedTech i) proposes a shift from design for usability to design for resilience, ii) aspires to reduce the need for service adaptation to technological constraints iii) ensures value of innovation at the time of product development, and iv) aims to stimulate discussion around the integration of pre- and post-market methods of assessment of medical technology.

Usability study of pH strips for nasogastric tube placement.

AIMS: (1) To model the process of use and usability of pH strips (2) to identify, through simulation studies, the likelihood of misreading pH strips, and to assess professional's acceptance, trust and perceived usability of pH strips. METHODS: This study was undertaken in four phases and used a mixed method approach (an audit, a semi-structured interview, a survey and simulation study). The three months audit was of 24 patients, the semi-structured interview was performed with 19 health professionals and informed the process of use of pH strips. A survey of 134 professionals and novices explored the likelihood of misinterpreting pH strips. Standardised questionnaires were used to assess professionals perceived usability, trust and acceptance of pH strip use in a simulated study. RESULTS: The audit found that in 45.7% of the cases aspiration could not be achieved, and that 54% of the NG-tube insertions required x-ray confirmation. None of those interviewed had received formal training on pH strips use. In the simulated study, participants made up to 11.15% errors in reading the strips with important implications for decision making regarding NG tube placement. No difference was identified between professionals and novices in their likelihood of misinterpreting the pH value of the strips. Whilst the overall experience of usage is poor (47.3%), health professionals gave a positive level of trust in both the interview (62.6%) and the survey (68.7%) and acceptance (interview group 65.1%, survey group 74.7%). They also reported anxiety in the use of strips (interview group 29.7%, survey group 49.7%). CONCLUSIONS: Significant errors occur when using pH strips in a simulated study. Manufacturers should consider developing new pH strips, specifically designed for bedside use, that are more usable and less likely to be misread.

Why you need to include human factors in clinical and empirical studies of in vitro point of care devices? Review and future perspectives.

Use of in-vitro point of care devices - intended as tests performed out of laboratories and near patient - is increasing in clinical environments. International standards indicate that interaction assessment should not end after the product release, yet human factors methods are frequently not included in clinical and empirical studies of these devices. Whilst the literature confirms some advantages of bed-side tests compared to those in laboratories there is a lack of knowledge of the risks associated with their use. This article provides a review of approaches applied by clinical researchers to model the use of in-vitro testing. Results suggest that only a few studies have explored human factor approaches. Furthermore, when researchers investigated people-device interaction these were predominantly limited to qualitative and not standardised approaches. The methodological failings and limitations of these studies, identified by us, demonstrate the growing need to integrate human factors methods in the medical field.

The abandonment of assistive technology in Italy: a survey of National Health Service users.

BACKGROUND: This study was an extension of research which began in the Umbria region in 2009. AIM: To investigate the extent to which assistive technology (AT) has been abandoned by users of the Italian National Health Service (ULHS) and the reasons for this. DESIGN: Observational study. SETTING: Users who received a hearing device (HD) or mobility device (MD) by ULHS between 2010 and 2013. POPULATION: 749 out of 3,791 ULHS users contacted via telephone completed the interview: 330 (44.06%) had a HD and 419 (55.94%) a MD. METHODS: Data were collected using a specially developed telephone interview questionnaire including the Italian version of the Quebec User Evaluation of Satisfaction with AT (QUEST 2.0) and Assistive Technology Use Follow-up Survey (ATUFS). RESULTS: 134 users (17.9%) were no longer using their assigned AT device within seven months of issue and 40% of this group reported that they had never used the device. Duration of use (for how long the AT device was used before abandonment) and satisfaction with service delivery did not predict AT abandonment. People who received a HD where more likely to abandon their device (22.4%) than those who received a MD (14.4%). CONCLUSIONS: Abandonment may be due to assignment of inappropriate devices or failure to meet user needs and expectations. These findings are consistent with previous data collected by Federici and Borsci in 2009. Utility of AT in use, reasons of abandonment, and importance of device and service satisfaction for the use or non-use of an AT are presented and discussed. CLINICAL REHABILITATION IMPACT: AT abandonment surveys provide useful information for modelling AT assessment and delivery process. The study confirms the relevance of person centredness approach for a successful AT assessment and delivery process.

Providing assistive technology in Italy: the perceived delivery process quality as affecting abandonment.

PURPOSE: The study brings together three aspects rarely observed at once in assistive technology (AT) surveys: (i) the assessment of user interaction/satisfaction with AT and service delivery, (ii) the motivational analysis of AT abandonment, and (iii) the management/design evaluation of AT delivery services. METHODS: 15 health professionals and 4 AT experts were involved in modelling and assessing four AT Local Health Delivery Service (Centres) in Italy through a SWOT analysis and a Cognitive Walkthrough. In addition 558 users of the same Centres were interviewed in a telephone survey to rate their satisfaction and AT use. RESULTS: The overall AT abandonment was equal to 19.09%. Different Centres' management strategies resulted in different percentages of AT disuse, with a range from 12.61% to 24.26%. A significant difference between the declared abandonment and the Centres' management strategies (p = 0.012) was identified. A strong effect on abandonment was also found due to professionals' procedures (p = 0.005) and follow-up systems (p = 0.002). CONCLUSIONS: The user experience of an AT is affected not only by the quality of the interaction with the AT, but also by the perceived quality of the Centres in support and follow-up. Implications for Rehabilitation AT abandonment surveys provide useful information for modelling AT assessment and delivery process. SWOT and Cognitive Walkthrough analyses have shown suitable methods for exploring limits and advantages in AT service delivery systems. The study confirms the relevance of person centredness for a successful AT assessment and delivery process.

How many testers are needed to assure the usability of medical devices?

Before releasing a product, manufacturers have to follow a regulatory framework and meet standards, producing reliable evidence that the device presents low levels of risk in use. There is, though, a gap between the needs of the manufacturers to conduct usability testing while managing their costs, and the requirements of authorities for representative evaluation data. A key issue here is the number of users that should complete this evaluation to provide confidence in a product's safety. This paper reviews the US FDA's indication that a sample composed of 15 participants per major group (or a minimum of 25 users) should be enough to identify 90-97% of the usability problems and argues that a more nuanced approach to determining sample size (which would also fit well with the FDA's own concerns) would be beneficial. The paper will show that there is no a priori cohort size that can guarantee a reliable assessment, a point stressed by the FDA in the appendices to its guidance, but that manufacturers can terminate the assessment when appropriate by using a specific approach - illustrated in this paper through a case study - called the 'Grounded Procedure'.

The Bootstrap Discovery Behaviour (BDB): a new outlook on usability evaluation.

The value of λ is one of the main issues debated in international usability studies. The debate is centred on the deficiencies of the mathematical return on investment model (ROI model) of Nielsen and Landauer (1993). The ROI model is discussed in order to identify the base of another model that, respecting Nielsen and Landauer's one, tries to consider a large number of variables for the estimation of the number of evaluators needed for an interface. Using the bootstrap model (Efron 1979), we can take into account: (a) the interface properties, as the properties at zero condition of evaluation and (b) the probability that the population discovery behaviour is represented by all the possible discovery behaviours of a sample. Our alternative model, named Bootstrap Discovery Behaviour (BDB), provides an alternative estimation of the number of experts and users needed for a usability evaluation. Two experimental groups of users and experts are involved in the evaluation of a website (http://www.serviziocivile.it). Applying the BDB model to the problems identified by the two groups, we found that 13 experts and 20 users are needed to identify 80% of usability problems, instead of 6 experts and 7 users required according to the estimation of the discovery likelihood provided by the ROI model. The consequence of the difference between the results of those models is that in following the BDB the costs of usability evaluation increase, although this is justified considering that the results obtained have the best probability of representing the entire population of experts and users.

Usability evaluation with screen reader users: a video presentation of the PCTA's experimental setting and rules.

In the study entitled "Web usability evaluation with screen reader users: Implementation of the Partial Concurrent Thinking Aloud technique" (Federici et al. 2010), we have proposed a modified protocol of usability evaluation technique for blind users, which integrates the features of the concurrent and the retrospective techniques. This new technique, called partial concurrent thinking aloud (PCTA), while respecting the properties of classic verbal protocols, overcomes the structural interference and the limits of concurrent and retrospective protocols when used with screen reader users. In order to facilitate understanding and acquisition of the PCTA's technique for practitioners and researchers, we have video recorded three different verbal protocols by visualizing five experimental sections. In the first two videos, we have compared a concurrent with a retrospective's verbal protocol of a sighted user, showing the difference of the verbalizations provided by the user in these two conditions. The third video shows the structural interference of the screen reader, during a blind user concurrent thinking aloud. In the last two videos, we show the difference of a blind user behaviour when PCTA or retrospective protocol is adopted. The videos clearly visualize the advantage of the PCTA use in respect of the two other protocols. In conclusion, the visualization of the PCTA technique confirms that this new verbal protocol promotes and guarantees a more user-driven usability assessment with disabled people, by better involving screen reader users, overcoming the structural interference and the limits of the concurrent and retrospective protocols.

Web usability evaluation with screen reader users: implementation of the partial concurrent thinking aloud technique.

A verbal protocol technique, adopted for a web usability evaluation, requires that the users are able to perform a double task: surfing and talking. Nevertheless, when blind users surf by using a screen reader and talk about the way they interact with the computer, the evaluation is influenced by a structural interference: users are forced to think aloud and listen to the screen reader at the same time. The aim of this study is to build up a verbal protocol technique for samples of visual impaired users in order to overcome the limits of concurrent and retrospective protocols. The technique we improved, called partial concurrent thinking aloud (PCTA), integrates a modified set of concurrent verbalization and retrospective analysis. One group of 6 blind users and another group of 6 sighted users evaluated the usability of a website using PCTA. By estimating the number of necessary users by the means of an asymptotic test, it was found out that the two groups had an equivalent ability of identifying usability problems, both over 80%. The result suggests that PCTA, while respecting the properties of classic verbal protocols, also allows to overcome the structural interference and the limits of concurrent and retrospective protocols when used with screen reader users. In this way, PCTA reduces the efficiency difference of usability evaluation between blind and sighted users.