Imaging biomarker roadmap for cancer studies.

Imaging biomarkers (IBs) are integral to the routine management of patients with cancer. IBs used daily in oncology include clinical TNM stage, objective response and left ventricular ejection fraction. Other CT, MRI, PET and ultrasonography biomarkers are used extensively in cancer research and drug development. New IBs need to be established either as useful tools for testing research hypotheses in clinical trials and research studies, or as clinical decision-making tools for use in healthcare, by crossing 'translational gaps' through validation and qualification. Important differences exist between IBs and biospecimen-derived biomarkers and, therefore, the development of IBs requires a tailored 'roadmap'. Recognizing this need, Cancer Research UK (CRUK) and the European Organisation for Research and Treatment of Cancer (EORTC) assembled experts to review, debate and summarize the challenges of IB validation and qualification. This consensus group has produced 14 key recommendations for accelerating the clinical translation of IBs, which highlight the role of parallel (rather than sequential) tracks of technical (assay) validation, biological/clinical validation and assessment of cost-effectiveness; the need for IB standardization and accreditation systems; the need to continually revisit IB precision; an alternative framework for biological/clinical validation of IBs; and the essential requirements for multicentre studies to qualify IBs for clinical use.

LUNGx Challenge for computerized lung nodule classification.

The purpose of this work is to describe the LUNGx Challenge for the computerized classification of lung nodules on diagnostic computed tomography (CT) scans as benign or malignant and report the performance of participants' computerized methods along with that of six radiologists who participated in an observer study performing the same Challenge task on the same dataset. The Challenge provided sets of calibration and testing scans, established a performance assessment process, and created an infrastructure for case dissemination and result submission. Ten groups applied their own methods to 73 lung nodules (37 benign and 36 malignant) that were selected to achieve approximate size matching between the two cohorts. Area under the receiver operating characteristic curve (AUC) values for these methods ranged from 0.50 to 0.68; only three methods performed statistically better than random guessing. The radiologists' AUC values ranged from 0.70 to 0.85; three radiologists performed statistically better than the best-performing computer method. The LUNGx Challenge compared the performance of computerized methods in the task of differentiating benign from malignant lung nodules on CT scans, placed in the context of the performance of radiologists on the same task. The continued public availability of the Challenge cases will provide a valuable resource for the medical imaging research community.

The Quantitative Imaging Network: NCI's Historical Perspective and Planned Goals.

The purpose of this editorial is to provide a brief history of National Institutes of Health National Cancer Institute (NCI) workshops as related to quantitative imaging within the oncology setting. The editorial will then focus on the recently supported NCI initiatives, including the Quantitative Imaging Network (QIN) initiative and its organizational structure, including planned research goals and deliverables. The publications in this issue of Translational Oncology come from many of the current members of this QIN research network.

Assessment of radiologist performance in the detection of lung nodules: dependence on the definition of "truth".

RATIONALE AND OBJECTIVES: Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the "truth"). The purpose of this study was to analyze (1) variability in the "truth" defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of "truth" in the context of lung nodule detection in computed tomographic (CT) scans. MATERIALS AND METHODS: Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules >or=3 mm in maximum diameter. Panel "truth" sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel "truth" sets. RESULTS: The number of "true" nodules in the different panel "truth" sets ranged from 15 to 89 (mean 49.8 +/- 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel "truth" sets for different panel "truth" conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case. CONCLUSIONS: Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of "truth" on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the "truth" established by other experienced thoracic radiologists.

Imaging as a Biomarker: Standards for Change Measurements in Therapy workshop summary.

Biomarkers are biological indicators of disease or therapeutic effects that can be measured by in vivo biomedical/molecular imaging, as well as other in vitro or laboratory methods. Recent work has shown that biomedical imaging can provide an early indication of drug response by use of x-ray, computed tomography (CT), positron-emission tomography/CT (PET/CT), or magnetic resonance imaging (MRI). There are three primary sources of uncertainty in using imaging as a biomarker: 1) the biological variability, 2) the variability associated with the clinicians interpreting the images, and 3) the physical measurement variability associated with image data collection and analysis across the same or different imaging platforms. Although biological variability is a large source of error, the physical uncertainty often significantly reduces the robustness of the imaging methods and the clinical decision tools required for quantitative measurement of therapy response over time. Physical and biological measurement uncertainties may be addressed prior to designing a clinical trial and thus help in reducing the case size and cost of a clinical trial associated with a drug submission to the U.S. Food and Drug Administration (FDA). The National Institute of Standards and Technology (NIST) has been approached over the last few years by several industry and medical stakeholders to address the physical sources of measurement uncertainty. NIST's initial research discovered that the characterization of measurement uncertainty poses many complex metrology and standardization problems on a scale that appears to need significant collaboration across the different medical imaging stakeholders. Many of the issues are similar to other scientific domains that NIST has addressed as part of its mission to provide metrology standards to enhance the competitiveness of U.S. industries. To better assess the measurement and standards needs for using imaging as a biomarker, NIST engaged leading representatives from many of the different imaging societies, the imaging, pharmaceutical and e-health and other health care stakeholders, as well as other key federal agencies (the National Institutes of Health Institutes and Centers [NIH ICs], and FDA) to organize and conduct a United States Measurement System (USMS) workshop: http://usms.nist.gov/workshops. The workshop entitled Imaging as a Biomarker: Standards for Change Measurements in Therapy, was thus held on September 14-15, 2006, at NIST in Gaithersburg, Maryland. (Workshop agenda, presentations. and final workshop report will be available at http://usms.nist.gov/workshops/bioimaging.htm.)

The Lung Image Database Consortium (LIDC): ensuring the integrity of expert-defined "truth".

RATIONALE AND OBJECTIVES: Computer-aided diagnostic (CAD) systems fundamentally require the opinions of expert human observers to establish "truth" for algorithm development, training, and testing. The integrity of this "truth," however, must be established before investigators commit to this "gold standard" as the basis for their research. The purpose of this study was to develop a quality assurance (QA) model as an integral component of the "truth" collection process concerning the location and spatial extent of lung nodules observed on computed tomography (CT) scans to be included in the Lung Image Database Consortium (LIDC) public database. MATERIALS AND METHODS: One hundred CT scans were interpreted by four radiologists through a two-phase process. For the first of these reads (the "blinded read phase"), radiologists independently identified and annotated lesions, assigning each to one of three categories: "nodule >or=3 mm," "nodule <3 mm," or "non-nodule >or=3 mm." For the second read (the "unblinded read phase"), the same radiologists independently evaluated the same CT scans, but with all of the annotations from the previously performed blinded reads presented; each radiologist could add to, edit, or delete their own marks; change the lesion category of their own marks; or leave their marks unchanged. The post-unblinded read set of marks was grouped into discrete nodules and subjected to the QA process, which consisted of identification of potential errors introduced during the complete image annotation process and correction of those errors. Seven categories of potential error were defined; any nodule with a mark that satisfied the criterion for one of these categories was referred to the radiologist who assigned that mark for either correction or confirmation that the mark was intentional. RESULTS: A total of 105 QA issues were identified across 45 (45.0%) of the 100 CT scans. Radiologist review resulted in modifications to 101 (96.2%) of these potential errors. Twenty-one lesions erroneously marked as lung nodules after the unblinded reads had this designation removed through the QA process. CONCLUSIONS: The establishment of "truth" must incorporate a QA process to guarantee the integrity of the datasets that will provide the basis for the development, training, and testing of CAD systems.

The Lung Image Database Consortium (LIDC) data collection process for nodule detection and annotation.

RATIONALE AND OBJECTIVES: The Lung Image Database Consortium (LIDC) is developing a publicly available database of thoracic computed tomography (CT) scans as a medical imaging research resource to promote the development of computer-aided detection or characterization of pulmonary nodules. To obtain the best estimate of the location and spatial extent of lung nodules, expert thoracic radiologists reviewed and annotated each scan. Because a consensus panel approach was neither feasible nor desirable, a unique two-phase, multicenter data collection process was developed to allow multiple radiologists at different centers to asynchronously review and annotate each CT scan. This data collection process was also intended to capture the variability among readers. MATERIALS AND METHODS: Four radiologists reviewed each scan using the following process. In the first or "blinded" phase, each radiologist reviewed the CT scan independently. In the second or "unblinded" review phase, results from all four blinded reviews were compiled and presented to each radiologist for a second review, allowing the radiologists to review their own annotations together with the annotations of the other radiologists. The results of each radiologist's unblinded review were compiled to form the final unblinded review. An XML-based message system was developed to communicate the results of each reading. RESULTS: This two-phase data collection process was designed, tested, and implemented across the LIDC. More than 500 CT scans have been read and annotated using this method by four expert readers; these scans either are currently publicly available at http://ncia.nci.nih.gov or will be in the near future. CONCLUSIONS: A unique data collection process was developed, tested, and implemented that allowed multiple readers at distributed sites to asynchronously review CT scans multiple times. This process captured the opinions of each reader regarding the location and spatial extent of lung nodules.

The Lung Image Database Consortium (LIDC): a comparison of different size metrics for pulmonary nodule measurements.

RATIONALE AND OBJECTIVES: The goal was to investigate the effects of choosing between different metrics in estimating the size of pulmonary nodules as a factor both of nodule characterization and of performance of computer aided detection systems, because the latter are always qualified with respect to a given size range of nodules. MATERIALS AND METHODS: This study used 265 whole-lung CT scans documented by the Lung Image Database Consortium (LIDC) using their protocol for nodule evaluation. Each inspected lesion was reviewed independently by four experienced radiologists who provided boundary markings for nodules larger than 3 mm. Four size metrics, based on the boundary markings, were considered: a unidimensional and two bidimensional measures on a single image slice and a volumetric measurement based on all the image slices. The radiologist boundaries were processed and those with four markings were analyzed to characterize the interradiologist variation, while those with at least one marking were used to examine the difference between the metrics. RESULTS: The processing of the annotations found 127 nodules marked by all of the four radiologists and an extended set of 518 nodules each having at least one observation with three-dimensional sizes ranging from 2.03 to 29.4 mm (average 7.05 mm, median 5.71 mm). A very high interobserver variation was observed for all these metrics: 95% of estimated standard deviations were in the following ranges for the three-dimensional, unidimensional, and two bidimensional size metrics, respectively (in mm): 0.49-1.25, 0.67-2.55, 0.78-2.11, and 0.96-2.69. Also, a very large difference among the metrics was observed: 0.95 probability-coverage region widths for the volume estimation conditional on unidimensional, and the two bidimensional size measurements of 10 mm were 7.32, 7.72, and 6.29 mm, respectively. CONCLUSIONS: The selection of data subsets for performance evaluation is highly impacted by the size metric choice. The LIDC plans to include a single size measure for each nodule in its database. This metric is not intended as a gold standard for nodule size; rather, it is intended to facilitate the selection of unique repeatable size limited nodule subsets.

The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans.

RATIONALE AND OBJECTIVES: The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on computed tomography (CT) scans and thereby to investigate variability in the establishment of the "truth" against which nodule-based studies are measured. MATERIALS AND METHODS: Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial "blinded read" phase, radiologists independently marked lesions they identified as "nodule >or=3 mm (diameter)," "nodule <3 mm," or "non-nodule >or=3 mm." During the subsequent "unblinded read" phase, the blinded read results of all four radiologists were revealed to each radiologist, who then independently reviewed their marks along with the anonymous marks of their colleagues; a radiologist's own marks then could be deleted, added, or left unchanged. This approach was developed to identify, as completely as possible, all nodules in a scan without requiring forced consensus. RESULTS: After the initial blinded read phase, 71 lesions received "nodule >or=3 mm" marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. After the unblinded reads, a total of 59 lesions were marked as "nodule >or=3 mm" by at least one radiologist. Twenty-seven (45.8%) of these lesions received such marks from all four radiologists, three (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist. CONCLUSION: The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules >or=3 mm. Nevertheless, substantial variability remains across radiologists in the task of lung nodule identification.

Evaluation of lung MDCT nodule annotation across radiologists and methods.

RATIONALE AND OBJECTIVES: Integral to the mission of the National Institutes of Health-sponsored Lung Imaging Database Consortium is the accurate definition of the spatial location of pulmonary nodules. Because the majority of small lung nodules are not resected, a reference standard from histopathology is generally unavailable. Thus assessing the source of variability in defining the spatial location of lung nodules by expert radiologists using different software tools as an alternative form of truth is necessary. MATERIALS AND METHODS: The relative differences in performance of six radiologists each applying three annotation methods to the task of defining the spatial extent of 23 different lung nodules were evaluated. The variability of radiologists' spatial definitions for a nodule was measured using both volumes and probability maps (p-map). Results were analyzed using a linear mixed-effects model that included nested random effects. RESULTS: Across the combination of all nodules, volume and p-map model parameters were found to be significant at P < .05 for all methods, all radiologists, and all second-order interactions except one. The radiologist and methods variables accounted for 15% and 3.5% of the total p-map variance, respectively, and 40.4% and 31.1% of the total volume variance, respectively. CONCLUSION: Radiologists represent the major source of variance as compared with drawing tools independent of drawing metric used. Although the random noise component is larger for the p-map analysis than for volume estimation, the p-map analysis appears to have more power to detect differences in radiologist-method combinations. The standard deviation of the volume measurement task appears to be proportional to nodule volume.

Lung image database consortium: developing a resource for the medical imaging research community.

To stimulate the advancement of computer-aided diagnostic (CAD) research for lung nodules in thoracic computed tomography (CT), the National Cancer Institute launched a cooperative effort known as the Lung Image Database Consortium (LIDC). The LIDC is composed of five academic institutions from across the United States that are working together to develop an image database that will serve as an international research resource for the development, training, and evaluation of CAD methods in the detection of lung nodules on CT scans. Prior to the collection of CT images and associated patient data, the LIDC has been engaged in a consensus process to identify, address, and resolve a host of challenging technical and clinical issues to provide a solid foundation for a scientifically robust database. These issues include the establishment of (a) a governing mission statement, (b) criteria to determine whether a CT scan is eligible for inclusion in the database, (c) an appropriate definition of the term qualifying nodule, (d) an appropriate definition of "truth" requirements, (e) a process model through which the database will be populated, and (f) a statistical framework to guide the application of assessment methods by users of the database. Through a consensus process in which careful planning and proper consideration of fundamental issues have been emphasized, the LIDC database is expected to provide a powerful resource for the medical imaging research community. This article is intended to share with the community the breadth and depth of these key issues.

Cancer imaging informatics workshop report from the Biomedical Imaging Program of the National Cancer Institute.

A group of experts on very large databases, quantitative imaging, data format standards development, image management and communications, and related technologies for cancer imaging met at a recent workshop sponsored by the BIP and discussed the key issues confronting this field. The BIP received recommendations regarding steps that can be taken to advance the technology and take advantage of the opportunities to improve collaboration and utility in cancer imaging. There are tremendous opportunities to change radically the way we use image information. These opportunities are most obvious in clinical research, in which we seek to advance the dissemination and use of cancer image data in research and practice. Important opportunities and new modes of information use are provided by supporting the entire "information cycle" of creation, dissemination, and collaboration in addition to image organization, access, presentation, and preservation. To learn more, visit the BIP Web site at www3 .cancer.gov/bip/steer_iasc.htm and join the image archive listserver at the NIH (ARCHIVE-COMM-L, available online at list.nihgov). These resources are open and available to all.