New Extensibility and Scripting Tools in the ImageJ Ecosystem.

ImageJ provides a framework for image processing across scientific domains while being fully open source. Over the years ImageJ has been substantially extended to support novel applications in scientific imaging as they emerge, particularly in the area of biological microscopy, with functionality made more accessible via the Fiji distribution of ImageJ. Within this software ecosystem, work has been done to extend the accessibility of ImageJ to utilize scripting, macros, and plugins in a variety of programming scenarios, e.g., from Groovy and Python and in Jupyter notebooks and cloud computing. We provide five protocols that demonstrate the extensibility of ImageJ for various workflows in image processing. We focus first on Fluorescence Lifetime Imaging Microscopy (FLIM) data, since this requires significant processing to provide quantitative insights into the microenvironments of cells. Second, we show how ImageJ can now be utilized for common image processing techniques, specifically image deconvolution and inversion, while highlighting the new, built-in features of ImageJ-particularly its capacity to run completely headless and the Ops matching feature that selects the optimal algorithm for a given function and data input, thereby enabling processing speedup. Collectively, these protocols can be used as a basis for automating biological image processing workflows. © 2021 Wiley Periodicals LLC. Basic Protocol 1: Using PyImageJ for FLIM data processing Alternate Protocol: Groovy FLIMJ in Jupyter Notebooks Basic Protocol 2: Using ImageJ Ops for image deconvolution Support Protocol 1: Using ImageJ Ops matching feature for image inversion Support Protocol 2: Headless ImageJ deconvolution.

The ImageJ ecosystem: Open-source software for image visualization, processing, and analysis.

For decades, biologists have relied on software to visualize and interpret imaging data. As techniques for acquiring images increase in complexity, resulting in larger multidimensional datasets, imaging software must adapt. ImageJ is an open-source image analysis software platform that has aided researchers with a variety of image analysis applications, driven mainly by engaged and collaborative user and developer communities. The close collaboration between programmers and users has resulted in adaptations to accommodate new challenges in image analysis that address the needs of ImageJ's diverse user base. ImageJ consists of many components, some relevant primarily for developers and a vast collection of user-centric plugins. It is available in many forms, including the widely used Fiji distribution. We refer to this entire ImageJ codebase and community as the ImageJ ecosystem. Here we review the core features of this ecosystem and highlight how ImageJ has responded to imaging technology advancements with new plugins and tools in recent years. These plugins and tools have been developed to address user needs in several areas such as visualization, segmentation, and tracking of biological entities in large, complex datasets. Moreover, new capabilities for deep learning are being added to ImageJ, reflecting a shift in the bioimage analysis community towards exploiting artificial intelligence. These new tools have been facilitated by profound architectural changes to the ImageJ core brought about by the ImageJ2 project. Therefore, we also discuss the contributions of ImageJ2 to enhancing multidimensional image processing and interoperability in the ImageJ ecosystem.

FLIMJ: An open-source ImageJ toolkit for fluorescence lifetime image data analysis.

In the field of fluorescence microscopy, there is continued demand for dynamic technologies that can exploit the complete information from every pixel of an image. One imaging technique with proven ability for yielding additional information from fluorescence imaging is Fluorescence Lifetime Imaging Microscopy (FLIM). FLIM allows for the measurement of how long a fluorophore stays in an excited energy state, and this measurement is affected by changes in its chemical microenvironment, such as proximity to other fluorophores, pH, and hydrophobic regions. This ability to provide information about the microenvironment has made FLIM a powerful tool for cellular imaging studies ranging from metabolic measurement to measuring distances between proteins. The increased use of FLIM has necessitated the development of computational tools for integrating FLIM analysis with image and data processing. To address this need, we have created FLIMJ, an ImageJ plugin and toolkit that allows for easy use and development of extensible image analysis workflows with FLIM data. Built on the FLIMLib decay curve fitting library and the ImageJ Ops framework, FLIMJ offers FLIM fitting routines with seamless integration with many other ImageJ components, and the ability to be extended to create complex FLIM analysis workflows. Building on ImageJ Ops also enables FLIMJ's routines to be used with Jupyter notebooks and integrate naturally with science-friendly programming in, e.g., Python and Groovy. We show the extensibility of FLIMJ in two analysis scenarios: lifetime-based image segmentation and image colocalization. We also validate the fitting routines by comparing them against industry FLIM analysis standards.

Integration of the ImageJ Ecosystem in the KNIME Analytics Platform.

Open-source software tools are often used for analysis of scientific image data due to their flexibility and transparency in dealing with rapidly evolving imaging technologies. The complex nature of image analysis problems frequently requires many tools to be used in conjunction, including image processing and analysis, data processing, machine learning and deep learning, statistical analysis of the results, visualization, correlation to heterogeneous but related data, and more. However, the development, and therefore application, of these computational tools is impeded by a lack of integration across platforms. Integration of tools goes beyond convenience, as it is impractical for one tool to anticipate and accommodate the current and future needs of every user. This problem is emphasized in the field of bioimage analysis, where various rapidly emerging methods are quickly being adopted by researchers. ImageJ is a popular open-source image analysis platform, with contributions from a global community resulting in hundreds of specialized routines for a wide array of scientific tasks. ImageJ's strength lies in its accessibility and extensibility, allowing researchers to easily improve the software to solve their image analysis tasks. However, ImageJ is not designed for development of complex end-to-end image analysis workflows. Scientists are often forced to create highly specialized and hard-to-reproduce scripts to orchestrate individual software fragments and cover the entire life-cycle of an analysis of an image dataset. KNIME Analytics Platform, a user-friendly data integration, analysis, and exploration workflow system, was designed to handle huge amounts of heterogeneous data in a platform-agnostic, computing environment and has been successful in meeting complex end-to-end demands in several communities, such as cheminformatics and mass spectrometry. Similar needs within the bioimage analysis community led to the creation of the KNIME Image Processing extension which integrates ImageJ into KNIME Analytics Platform, enabling researchers to develop reproducible and scalable workflows, integrating a diverse range of analysis tools. Here we present how users and developers alike can leverage the ImageJ ecosystem via the KNIME Image Processing extension to provide robust and extensible image analysis within KNIME workflows. We illustrate the benefits of this integration with examples, as well as representative scientific use cases.

Scientific Community Image Forum: A discussion forum for scientific image software.

Forums and email lists play a major role in assisting scientists in using software. Previously, each open-source bioimaging software package had its own distinct forum or email list. Although each provided access to experts from various software teams, this fragmentation resulted in many scientists not knowing where to begin with their projects. Thus, the scientific imaging community lacked a central platform where solutions could be discussed in an open, software-independent manner. In response, we introduce the Scientific Community Image Forum, where users can pose software-related questions about digital image analysis, acquisition, and data management.

Assessing microscope image focus quality with deep learning.

BACKGROUND: Large image datasets acquired on automated microscopes typically have some fraction of low quality, out-of-focus images, despite the use of hardware autofocus systems. Identification of these images using automated image analysis with high accuracy is important for obtaining a clean, unbiased image dataset. Complicating this task is the fact that image focus quality is only well-defined in foreground regions of images, and as a result, most previous approaches only enable a computation of the relative difference in quality between two or more images, rather than an absolute measure of quality. RESULTS: We present a deep neural network model capable of predicting an absolute measure of image focus on a single image in isolation, without any user-specified parameters. The model operates at the image-patch level, and also outputs a measure of prediction certainty, enabling interpretable predictions. The model was trained on only 384 in-focus Hoechst (nuclei) stain images of U2OS cells, which were synthetically defocused to one of 11 absolute defocus levels during training. The trained model can generalize on previously unseen real Hoechst stain images, identifying the absolute image focus to within one defocus level (approximately 3 pixel blur diameter difference) with 95% accuracy. On a simpler binary in/out-of-focus classification task, the trained model outperforms previous approaches on both Hoechst and Phalloidin (actin) stain images (F-scores of 0.89 and 0.86, respectively over 0.84 and 0.83), despite only having been presented Hoechst stain images during training. Lastly, we observe qualitatively that the model generalizes to two additional stains, Hoechst and Tubulin, of an unseen cell type (Human MCF-7) acquired on a different instrument. CONCLUSIONS: Our deep neural network enables classification of out-of-focus microscope images with both higher accuracy and greater precision than previous approaches via interpretable patch-level focus and certainty predictions. The use of synthetically defocused images precludes the need for a manually annotated training dataset. The model also generalizes to different image and cell types. The framework for model training and image prediction is available as a free software library and the pre-trained model is available for immediate use in Fiji (ImageJ) and CellProfiler.

FunImageJ: a Lisp framework for scientific image processing.

Summary: FunImageJ is a Lisp framework for scientific image processing built upon the ImageJ software ecosystem. The framework provides a natural functional-style for programming, while accounting for the performance requirements necessary in big data processing commonly encountered in biological image analysis. Availability and implementation: Freely available plugin to Fiji (http://fiji.sc/#download). Installation and use instructions available at http://imagej.net/FunImageJ. Contact: kharrington@uidaho.edu. Supplementary information: Supplementary data are available at Bioinformatics online.

ImageJ2: ImageJ for the next generation of scientific image data.

BACKGROUND: ImageJ is an image analysis program extensively used in the biological sciences and beyond. Due to its ease of use, recordable macro language, and extensible plug-in architecture, ImageJ enjoys contributions from non-programmers, amateur programmers, and professional developers alike. Enabling such a diversity of contributors has resulted in a large community that spans the biological and physical sciences. However, a rapidly growing user base, diverging plugin suites, and technical limitations have revealed a clear need for a concerted software engineering effort to support emerging imaging paradigms, to ensure the software's ability to handle the requirements of modern science. RESULTS: We rewrote the entire ImageJ codebase, engineering a redesigned plugin mechanism intended to facilitate extensibility at every level, with the goal of creating a more powerful tool that continues to serve the existing community while addressing a wider range of scientific requirements. This next-generation ImageJ, called "ImageJ2" in places where the distinction matters, provides a host of new functionality. It separates concerns, fully decoupling the data model from the user interface. It emphasizes integration with external applications to maximize interoperability. Its robust new plugin framework allows everything from image formats, to scripting languages, to visualization to be extended by the community. The redesigned data model supports arbitrarily large, N-dimensional datasets, which are increasingly common in modern image acquisition. Despite the scope of these changes, backwards compatibility is maintained such that this new functionality can be seamlessly integrated with the classic ImageJ interface, allowing users and developers to migrate to these new methods at their own pace. CONCLUSIONS: Scientific imaging benefits from open-source programs that advance new method development and deployment to a diverse audience. ImageJ has continuously evolved with this idea in mind; however, new and emerging scientific requirements have posed corresponding challenges for ImageJ's development. The described improvements provide a framework engineered for flexibility, intended to support these requirements as well as accommodate future needs. Future efforts will focus on implementing new algorithms in this framework and expanding collaborations with other popular scientific software suites.

ImageJ-MATLAB: a bidirectional framework for scientific image analysis interoperability.

Summary: ImageJ-MATLAB is a lightweight Java library facilitating bi-directional interoperability between MATLAB and ImageJ. By defining a standard for translation between matrix and image data structures, researchers are empowered to select the best tool for their image-analysis tasks. Availability and Implementation: Freely available extension to ImageJ2 ( http://imagej.net/Downloads ). Installation and use instructions available at http://imagej.net/MATLAB_Scripting. Tested with ImageJ 2.0.0-rc-54 , Java 1.8.0_66 and MATLAB R2015b. Contact: eliceiri@wisc.edu. Supplementary information: Supplementary data are available at Bioinformatics online.

Quantitating the cell: turning images into numbers with ImageJ.

Modern biological research particularly in the fields of developmental and cell biology has been transformed by the rapid evolution of the light microscope. The light microscope, long a mainstay of the experimental biologist, is now used for a wide array of biological experimental scenarios and sample types. Much of the great developments in advanced biological imaging have been driven by the digital imaging revolution with powerful processors and algorithms. In particular, this combination of advanced imaging and computational analysis has resulted in the drive of the modern biologist to not only visually inspect dynamic phenomena, but to quantify the involved processes. This need to quantitate images has become a major thrust within the bioimaging community and requires extensible and accessible image processing routines with corresponding intuitive software packages. Novel algorithms both made specifically for light microscopy or adapted from other fields, such as astronomy, are available to biologists, but often in a form that is inaccessible for a number of reasons ranging from data input issues, usability and training concerns, and accessibility and output limitations. The biological community has responded to this need by developing open source software packages that are freely available and provide access to image processing routines. One of the most prominent is the open-source image package ImageJ. In this review, we give an overview of prominent imaging processing approaches in ImageJ that we think are of particular interest for biological imaging and that illustrate the functionality of ImageJ and other open source image analysis software. WIREs Dev Biol 2017, 6:e260. doi: 10.1002/wdev.260 For further resources related to this article, please visit the WIREs website.

SCIFIO: an extensible framework to support scientific image formats.

BACKGROUND: No gold standard exists in the world of scientific image acquisition; a proliferation of instruments each with its own proprietary data format has made out-of-the-box sharing of that data nearly impossible. In the field of light microscopy, the Bio-Formats library was designed to translate such proprietary data formats to a common, open-source schema, enabling sharing and reproduction of scientific results. While Bio-Formats has proved successful for microscopy images, the greater scientific community was lacking a domain-independent framework for format translation. RESULTS: SCIFIO (SCientific Image Format Input and Output) is presented as a freely available, open-source library unifying the mechanisms of reading and writing image data. The core of SCIFIO is its modular definition of formats, the design of which clearly outlines the components of image I/O to encourage extensibility, facilitated by the dynamic discovery of the SciJava plugin framework. SCIFIO is structured to support coexistence of multiple domain-specific open exchange formats, such as Bio-Formats' OME-TIFF, within a unified environment. CONCLUSIONS: SCIFIO is a freely available software library developed to standardize the process of reading and writing scientific image formats.

The ImageJ ecosystem: An open platform for biomedical image analysis.

Technology in microscopy advances rapidly, enabling increasingly affordable, faster, and more precise quantitative biomedical imaging, which necessitates correspondingly more-advanced image processing and analysis techniques. A wide range of software is available-from commercial to academic, special-purpose to Swiss army knife, small to large-but a key characteristic of software that is suitable for scientific inquiry is its accessibility. Open-source software is ideal for scientific endeavors because it can be freely inspected, modified, and redistributed; in particular, the open-software platform ImageJ has had a huge impact on the life sciences, and continues to do so. From its inception, ImageJ has grown significantly due largely to being freely available and its vibrant and helpful user community. Scientists as diverse as interested hobbyists, technical assistants, students, scientific staff, and advanced biology researchers use ImageJ on a daily basis, and exchange knowledge via its dedicated mailing list. Uses of ImageJ range from data visualization and teaching to advanced image processing and statistical analysis. The software's extensibility continues to attract biologists at all career stages as well as computer scientists who wish to effectively implement specific image-processing algorithms. In this review, we use the ImageJ project as a case study of how open-source software fosters its suites of software tools, making multitudes of image-analysis technology easily accessible to the scientific community. We specifically explore what makes ImageJ so popular, how it impacts the life sciences, how it inspires other projects, and how it is self-influenced by coevolving projects within the ImageJ ecosystem.

Multiphoton flow cytometry to assess intrinsic and extrinsic fluorescence in cellular aggregates: applications to stem cells.

Detection and tracking of stem cell state are difficult due to insufficient means for rapidly screening cell state in a noninvasive manner. This challenge is compounded when stem cells are cultured in aggregates or three-dimensional (3D) constructs because living cells in this form are difficult to analyze without disrupting cellular contacts. Multiphoton laser scanning microscopy is uniquely suited to analyze 3D structures due to the broad tunability of excitation sources, deep sectioning capacity, and minimal phototoxicity but is throughput limited. A novel multiphoton fluorescence excitation flow cytometry (MPFC) instrument could be used to accurately probe cells in the interior of multicell aggregates or tissue constructs in an enhanced-throughput manner and measure corresponding fluorescent properties. By exciting endogenous fluorophores as intrinsic biomarkers or exciting extrinsic reporter molecules, the properties of cells in aggregates can be understood while the viable cellular aggregates are maintained. Here we introduce a first generation MPFC system and show appropriate speed and accuracy of image capture and measured fluorescence intensity, including intrinsic fluorescence intensity. Thus, this novel instrument enables rapid characterization of stem cells and corresponding aggregates in a noninvasive manner and could dramatically transform how stem cells are studied in the laboratory and utilized in the clinic.

Metadata matters: access to image data in the real world.

Data sharing is important in the biological sciences to prevent duplication of effort, to promote scientific integrity, and to facilitate and disseminate scientific discovery. Sharing requires centralized repositories, and submission to and utility of these resources require common data formats. This is particularly challenging for multidimensional microscopy image data, which are acquired from a variety of platforms with a myriad of proprietary file formats (PFFs). In this paper, we describe an open standard format that we have developed for microscopy image data. We call on the community to use open image data standards and to insist that all imaging platforms support these file formats. This will build the foundation for an open image data repository.

Nonlinear optical microscopy and computational analysis of intrinsic signatures in breast cancer.

Recently, new non-invasive imaging methods have been developed and applied to cellular and animal mammary models that have enabled breast cancer researchers to track key players and events in mammary metastasis. Noninvasive nonlinear optical methods such as multiphoton laser scanning microscopy (MPLSM), Fluorescence Lifetime Microscopy (FLIM) and second harmonic generation (SHG) imaging provide an unrivaled ability for obtaining high-resolution images from deep within tissue that can be exploited in the quest to understand breast cancer progression. These optical methods can add greatly to our knowledge of cancer progression by allowing key processes to be non-invasively imaged such as metabolism (on the basis of free and bound NADH detection via FLIM) and interactions with the extracellular matrix (SHG imaging of collagen). In this short application note we present a survey of our latest optical and computational efforts to study intrinsic fluorescence in breast cancer models. In particular we present the latest development in our SLIM Plotter application, an open source visualization program for interactive visualization and inspection of combined spectral lifetime (SLIM) data.

Nonlinear optical imaging and spectral-lifetime computational analysis of endogenous and exogenous fluorophores in breast cancer.

Multiphoton laser scanning microscopy (MPLSM) utilizing techniques such as multiphoton excitation (MPE), second harmonic generation (SHG), and multiphoton fluorescence lifetime imaging and spectral lifetime imaging (FLIM and SLIM, respectively) are greatly expanding the degree of information obtainable with optical imaging in biomedical research. The application of these nonlinear optical approaches to the study of breast cancer holds particular promise. These noninvasive, multidimensional techniques are well suited to image exogenous fluorophores that allow relevant questions regarding protein localization and signaling to be addressed both in vivo and in vitro. Furthermore, MPLSM imaging of endogenous signals from collagen and fluorophores such as nicotinamide adenine dinucleotide (NADH) or flavin adenine dinucleotide (FAD), address important questions regarding the tumor-stromal interaction and the physiologic state of the cell. We demonstrate the utility of multimodal MPE/SHG/FLIM for imaging both exogenous and/or endogenous fluorophores in mammary tumors or relevant 3-D systems. Using SLIM, we present a method for imaging and differentiating signals from multiple fluorophores that can have overlapping spectra via SLIM Plotter-a computational tool for visualizing and analyzing large spectral-lifetime data sets.

Collagen density promotes mammary tumor initiation and progression.

BACKGROUND: Mammographically dense breast tissue is one of the greatest risk factors for developing breast carcinoma. Despite the strong clinical correlation, breast density has not been causally linked to tumorigenesis, largely because no animal model has existed for studying breast tissue density. Importantly, regions of high breast density are associated with increased stromal collagen. Thus, the influence of the extracellular matrix on breast carcinoma development and the underlying molecular mechanisms are not understood. METHODS: To study the effects of collagen density on mammary tumor formation and progression, we utilized a bi-transgenic tumor model with increased stromal collagen in mouse mammary tissue. Imaging of the tumors and tumor-stromal interface in live tumor tissue was performed with multiphoton laser-scanning microscopy to generate multiphoton excitation and spectrally resolved fluorescent lifetimes of endogenous fluorophores. Second harmonic generation was utilized to image stromal collagen. RESULTS: Herein we demonstrate that increased stromal collagen in mouse mammary tissue significantly increases tumor formation approximately three-fold (p < 0.00001) and results in a significantly more invasive phenotype with approximately three times more lung metastasis (p < 0.05). Furthermore, the increased invasive phenotype of tumor cells that arose within collagen-dense mammary tissues remains after tumor explants are cultured within reconstituted three-dimensional collagen gels. To better understand this behavior we imaged live tumors using nonlinear optical imaging approaches to demonstrate that local invasion is facilitated by stromal collagen re-organization and that this behavior is significantly increased in collagen-dense tissues. In addition, using multiphoton fluorescence and spectral lifetime imaging we identify a metabolic signature for flavin adenine dinucleotide, with increased fluorescent intensity and lifetime, in invading metastatic cells. CONCLUSION: This study provides the first data causally linking increased stromal collagen to mammary tumor formation and metastasis, and demonstrates that fundamental differences arise and persist in epithelial tumor cells that progressed within collagen-dense microenvironments. Furthermore, the imaging techniques and signature identified in this work may provide useful diagnostic tools to rapidly assess fresh tissue biopsies.

Visualization approaches for multidimensional biological image data.

Effective data analysis of the modern biological microscopy data set often necessitates a variety of different analysis strategies, and this often means the biologist may need to use a combination of software tools both commercial and often-times open source. To facilitate this process, there needs to be knowledge of what the approaches are and also practical ways of sharing this data in a nonproprietary way. Thus, for users of open source and commercial software, it is important to have common approaches for multidimensional data analysis that can be run in different software packages and still be effectively compared. Projects like the Open Microscopy Environment, which aim to allow data sharing between open source client tools like ImageJ and VisBio, and commercial packages like Volocity and Imaris via the XML data model are a needed first step in providing a framework or infrastructure for microscopy analysis. As the field has gotten more quantitative in its approaches, this need has only increased with the necessity of having a way to represent key attributes of the data in an open manner.