Automated segmentation of liver segment on portal venous phase MR images using a 3D convolutional neural network.

OBJECTIVE: We aim to develop and validate a three-dimensional convolutional neural network (3D-CNN) model for automatic liver segment segmentation on MRI images. METHODS: This retrospective study evaluated an automated method using a deep neural network that was trained, validated, and tested with 367, 157, and 158 portal venous phase MR images, respectively. The Dice similarity coefficient (DSC), mean surface distance (MSD), Hausdorff distance (HD), and volume ratio (RV) were used to quantitatively measure the accuracy of segmentation. The time consumed for model and manual segmentation was also compared. In addition, the model was applied to 100 consecutive cases from real clinical scenario for a qualitative evaluation and indirect evaluation. RESULTS: In quantitative evaluation, the model achieved high accuracy for DSC, MSD, HD and RV (0.920, 3.34, 3.61 and 1.01, respectively). Compared to manual segmentation, the automated method reduced the segmentation time from 26 min to 8 s. In qualitative evaluation, the segmentation quality was rated as good in 79% of the cases, moderate in 15% and poor in 6%. In indirect evaluation, 93.4% (99/106) of lesions could be assigned to the correct segment by only referring to the results from automated segmentation. CONCLUSION: The proposed model may serve as an effective tool for automated anatomical region annotation of the liver on MRI images.

Topic-Oriented Image Captioning Based on Order-Embedding.

We present an image captioning framework that generates captions under a given topic. The topic candidates are extracted from the caption corpus. A given image's topics are then selected from these candidates by a CNN-based multi-label classifier. The input to the caption generation model is an image-topic pair, and the output is a caption of the image. For this purpose, a cross-modal embedding method is learned for the images, topics, and captions. In the proposed framework, the topic, caption, and image are organized in a hierarchical structure, which is preserved in the embedding space by using the order-embedding method. The caption embedding is upper bounded by the corresponding image embedding and lower bounded by the topic embedding. The lower bound pushes the images and captions about the same topic closer together in the embedding space. A bidirectional caption-image retrieval task is conducted on the learned embedding space and achieves the state-of-the-art performance on the MS-COCO and Flickr30K datasets, demonstrating the effectiveness of the embedding method. To generate a caption for an image, an embedding vector is sampled from the region bounded by the embeddings of the image and the topic, then a language model decodes it to a sentence as the output. The lower bound set by the topic shrinks the output space of the language model, which may help the model to learn to match images and captions better. Experiments on the image captioning task on the MS-COCO and Flickr30K datasets validate the usefulness of this framework by showing that the different given topics can lead to different captions describing specific aspects of the given image and that the quality of generated captions is higher than the control model without a topic as input. In addition, the proposed method is competitive with many state-of-the-art methods in terms of standard evaluation metrics.