Defining decision thresholds for judgments on health benefits and harms using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Evidence to Decision (EtD) frameworks: a protocol for a randomised methodological study (GRADE-THRESHOLD).

INTRODUCTION: The Grading of Recommendations Assessment, Development and Evaluation (GRADE) and similar Evidence to Decision (EtD) frameworks require its users to judge how substantial the effects of interventions are on desirable and undesirable people-important health outcomes. However, decision thresholds (DTs) that could help understand the magnitude of intervention effects and serve as reference for interpretation of findings are not yet available.The objective of this study is an approach to derive and use DTs for EtD judgments about the magnitude of health benefits and harms. We hypothesise that approximate DTs could have the ability to discriminate between the existing four categories of EtD judgments (Trivial, Small, Moderate, Large), support panels of decision-makers in their work, and promote consistency and transparency in judgments. METHODS AND ANALYSIS: We will conduct a methodological randomised controlled trial to collect the data that allow deriving the DTs. We will invite clinicians, epidemiologists, decision scientists, health research methodologists, experts in Health Technology Assessment (HTA), members of guideline development groups and the public to participate in the trial. Then, we will investigate the validity of our DTs by measuring the agreement between judgments that were made in the past by guideline panels and the judgments that our DTs approach would suggest if applied on the same guideline data. ETHICS AND DISSEMINATION: The Hamilton Integrated Research Ethics Board reviewed this study as a quality improvement study and determined that it requires no further consent. Survey participants will be required to read a consent statement in order to participate in this study at the beginning of the trial. This statement reads: You are being invited to participate in a research project which aims to identify indicative DTs that could assist users of the GRADE EtD frameworks in making judgments. Your input will be used in determining these indicative thresholds. By completing this survey, you provide consent that the anonymised data collected will be used for the research study and to be summarised in aggregate in publication and electronic tools. PROTOCOL REGISTRATION NUMBER: NCT05237635.

Which factors may determine the necessary and feasible type of effectiveness evidence? A mixed methods approach to develop an instrument to help coverage decision-makers.

OBJECTIVES: Reimbursement decisions require evidence of effectiveness and, in general, a blinded randomised controlled trial (RCT) is the preferred study design to provide it. However, there are situations where a cohort study, or even patient series, can be deemed acceptable. The aim of this study was to develop an instrument that first examines which study characteristics of a blinded RCT are necessary, and then, if particular characteristics are considered necessary, examines whether these characteristics are feasible. DESIGN: We retrospectively studied 22 interventions from 20 reimbursement reports concerning medical specialist care made by the Dutch National Health Care Institute (ZIN) to identify any factors that influenced the necessity and feasibility of blinded RCTs, and their constituent study characteristics, that is, blinding, randomisation and a control group. A literature review was performed to identify additional factors. Additional expertise was included by interviewing eight experts in epidemiology, medicine and ethics. The resulting instrument was called the FIT instrument (Feasible Information Trajectory), and was prospectively validated using three consecutive reimbursement reports. RESULTS: (Blinded) RCT evidence was lacking in 5 of 11 positive reimbursement decisions and 3 of 11 negative decisions. In the reimbursement reports, we found no empirical evidence supporting situations where a blinded RCT is unnecessary. The literature also revealed few arguments against the necessity of a blinded RCT. In contrast, many factors influencing the feasibility of randomisation, a control group and blinding, were found in the reimbursement reports and the literature; for example, when a patient population is too small or when an intervention is common practice, randomisation will be hindered. CONCLUSIONS: Policy regarding the necessity and feasibility of different types of evidence of effectiveness would benefit from systematic guidance. The FIT instrument has the potential to support transparent, reproducible and well-founded decisions on appropriate evidence of effectiveness in medical specialist care.

['Fitting' evidence preferable when evaluating effectiveness of interventions]. / Passend bewijs bij het bepalen van effectiviteit van interventies.

The Dutch Health Care Insurance Board (CVZ) and the Institute for Medical Technology Assessment have developed a questionnaire to help assessors to formulate an answer to the question of which evidence fits best when evaluating the effectiveness of interventions by medical specialists. The Feasible Information Trajectory (FIT) questionnaire is based on the idea that the clinical setting defines the attainable study characteristics and thus possibly available evidence. The FIT questionnaire focuses on study characteristics (randomization, blinding and control groups) and not on study type (e.g. RCT, cohort) as is common within evidence-hierarchy grading systems. Together these attainable characteristics define the most -fitting evidence. The questionnaire should be filled in prior to the literature assessment. Subsequently the most-fitting evidence profile can be modified on the basis of a literature assessment and possibly arguments put forward by healthcare professionals. Filling out the FIT questionnaire is therefore not a static exercise but is part of the search for all relevant arguments in the assessment of effectiveness.

The influence of picture size on recognition and exploratory behaviour in raised-line drawings.

We demonstrate the influence of picture size on haptic recognition and exploratory behaviour. The stimuli were raised-line drawings of everyday objects. Participants were instructed to think aloud during haptic exploration of the pictures. We measured the delay between initial correct speculation and final correct response. The results indicate that picture size influences accuracy but not response latency: large drawings are recognised more often but not faster. By analysing video recordings of the experiment we found that two-handed exploration increases when picture size increases and that, on average, 83% of the exploration time involves the use of two hands. The thinking-aloud data showed that the average time difference between the initial correct speculation and final correct response amounted to 23% of the total reaction time. We discuss our results with respect to the design of tactile aids and the ecological validity of single-finger exploration.

Look what I have felt: unidentified haptic line drawings are identified after sketching.

The difficulty that observers experience when trying to identify a raised line drawing by touch is still largely unexplained. In this article, we show that observers who are unable to haptically identify a raised line drawing are suddenly able to do so after they have sketched on paper what they have in their mind. We conducted three experiments: first of all we show that this effect is robust; in the second experiment, we show that identification-after-sketching is caused by visual inspection of the sketch, and not caused by feedback in general; and in the third we show that sketches which were identified by the observers who produced them, were also identified by completely naive viewers. These experiments demonstrate that during raised line drawing identification the mental capacities required to interpret the stimulus seem to be inadequate: although enough pictorial information was present to produce a sketch which could even be identified by naive viewers, the stimulus could not be identified by haptic and mental processing alone. Furthermore, we investigated whether increasing the haptic perceptual field by using two hands instead of one hand had an influence on identification performance. We did indeed find that using two hands significantly increased identification. We use both results to discuss the underlying mechanisms of haptic raised line drawing identification.

Ambiguous figures: living versus nonliving objects.

Eleven series of figures were studied, each series ranging from one extreme interpretation via five ambiguous intermediates to a second extreme interpretation. Triplets consisting of an ambiguous exemplar in the middle flanked on the left and right by its two extreme interpretations were presented to large groups of subjects. The initial aim was to establish the levels of perceptual ambiguity of each exemplar in a series, and normative data on the ambiguous figures are provided for future reference and use. However, several biases were encountered and these were examined in more detail. In experiment 1 the subject's task was to compare the middle figure with the flankers and draw an arrow from the middle figure towards the flanking extreme they judged the most similar. Here, an overall preference for the left extreme was found. Therefore the instructions were reversed in experiment 2; flankers had to be compared with the middle figure. The preference for the left extreme remained for figures of living objects, but for nonliving objects the preference switched to the right extreme. To do away with any effect of the arrows, in experiment 3 subjects were divided into two groups each receiving different instructions and were asked to circle one of the extremes. However, the pattern of biases remained the same. The bias found with figures of living objects may be explained on the basis of top-down processes. For nonliving figures, an hypothesis based on bottom-up processes like neural fatigue was considered but rejected.