Cancer Diagnosing
2017 - 2019
Ning Li, Haopeng Liu, Bin Qiu, Shijun Zhao, Kungang Li, Jie He, Shizun Liu
with Department of Thoracic surgery, Cancer Hospital, Chinese Academic of Medical Science, Beijing, China.
Role:
Research and Development
Design Tools:
Python, Pytorch, CAD, ImgJ, Photoshop
Brief:
This project proposes a novel and efficient method to build a Computer-Aided Diagnoses (CAD) system for lung nodule detection based on Computed Tomography (CT). This task was treated as an Object Detection on Video (VID) problem by imitating how a radiologist reads CT scans. A lung nodule detector was trained to automatically learn nodule features from still images to detect lung nodule candidates with both high recall and accuracy. Unlike previous work which used 3-dimensional information around the nodule to reduce false positives, we propose two simple but efficient methods, Multi-slice propagation (MSP) and Motionless-guide suppression (MLGS), which analyze sequence information of CT scans to reduce false negatives and suppress false positives. We evaluated our method in open-source LUNA16 dataset which contains 888 CT scans, and obtained state-of-the-art result (Free-Response Receiver Operating Characteristic score of 0.892) with detection speed (end to end within 20 seconds per patient on a single NVidia GTX 1080) much higher than existing methods.
Reduce
Recover
Top: Multi-slice propagation. Bottom: Motionless-guide suppression
About 30-40% pulmonary nodule might have connection with lung cancer. Patients who are diagnosed at an early stage of lung cancer have a significantly better survival rate than at a later stage. 5-years survival rate drops from over 80% when the lung cancer is detected in stage I to 1% when it is detected in stage IV. Hence, it is of high importance to detect the nodule in early stage.
Many endeavors have been made to use CAD system to distinguish lung nodule from other similar tissues such as blood vessel and weasand. A representative approach attempted to merge the adjacent 3 CT slices to detect lung nodule in 2D images with high recall, and reduced false positives using the 3D information around the candidate nodule. In another work, the author observed a nodule candidate from 9 different directions and provided diagnosis by synthesizing all the information. A third method is to utilize the 3D convolution network by directly taking a 3D image of the lung as input and outputting the nodule position and probability. These methods are computationally intensive even with high-end hardware (e.g., cluster of Nvidia Titan X), impeding their affordable real-world application.
In practice, radiologists observe each CT slice together with adjacent slices. Analysis of the nodule characteristics includes size, density, edges, the relationship between the surrounding tissue, and bronchial and vascular relationship. Thanks to advances in CT technology, coronal and sagittal pattern reconstruction is also applied to the clinic, providing more information to the radiologist. However, ultimately, 3-dimensional graphics need to be formed in the minds of radiologists, who then make the final judgment; this usually requires professional training and years of experience.
Because the workflow of radiologists is similar to detecting an object in video, we borrowed the main idea of VID and trained a reliable nodule detector to detect the nodule candidates on still images. Thereafter, we analyzed the correlation of detections on adjacent slices, reduced the false negatives and suppressed false positives based on the sequence information.
Obviously, the performance of still image detector significantly influences the detection result. In recent years, the lung nodule detection algorithm has been transforming from “feature engineering” to “network engineering”. In contrast to hand-designed features (e.g., intensity thresholding, shape curvedness, and mathematical morphology), features learned by a neural network from large-scale data (e.g. LIDC-IDRI) require minimal human involvement but show state-of-the-art performance in many medical image analysis applications. Therefore, we made efforts on designing innovative neural networks to learn rich semantic representations. We improved the state-of-the-art region-free object detection algorithm Single Shot Multibox Detector (SSD)by stacking multi-level and multi-resolution features from deep convolution network (ConvNet), which can overcome the inefficiency of SSD in small object detection. Experiments showed that our method was efficient to detect small lung nodules.
We proposed two simple but efficient algorithms called Multi-slice propagation (MSP) and Motionless-guide suppression (MLGS) to reduce false negatives and false positives, respectively. As shown in above figure, MSP recovers the missing detections by propagating the detection result to adjacent slice. MLGS builds a series of features to describe nodule motion and reduces false positives with Random Forest algorithm.
This work has three innovations:
(1) We designed an efficient nodule detector on 2D image based on the SSD detector, which takes advantage of multi-level features of the ConvNet and overcomes the SSD’s poor performance in detecting small object.
(2) We proposed two novel algorithms, MSP and MLGS, to reduce false negatives and false positives by incorporating the temporal and context information from adjacent slices.
(3) We investigated the main difference between natural and CT image processing, providing guidelines to improve the model performance for other researchers.
We have experimented several methods, and here we firstly compare the main difference between lung nodule detection tsk and VID task. From this, we have built our the framework of method by using still image detector. The methods of using multi-context information and motion information of reduce false negative and false detections.
A. VID and lung nodule detection task:
The ImageNet object detection from video (VID) task requires participants to localize 30 different targets in the video frames. For each video frame, the algorithm needs to produce a set of annotations (fi, ci, si, bi ) of frame index fi, class label ci, confidence score si and bounding box bi.
The lung nodule detection task is to output a set of annotations (xi, yi, zi, di, si) of the lung nodule center position ( xi, yi, zi), diameter di and confidence si. Since the task is similar to the VID task, we attempted to apply similar technologies to lung nodule detection task.
The main differences of lung nodule detection and VID task are:
(1) Unlike VID task, we cannot train a reliable still image detector using only a single CT slice, because it cannot provide enough information to distinguish a nodule from other tissues.
(2) Detectors designed for VID cannot be directly used for lung nodule detection, because the detector backbone (e.g., ResNet) for natural images may not fit the CT images.
(3) Motions exist in VID objects so that an object may be present in different locations in different frames; on the other hand, the lung nodule is motionless and is at nearly the same location in adjacent slices.
The CAD system for lung nodule detector is shown above.
The system consists of three main parts:
(1) Still image detector, that predicts lung nodules in multi-scale convolutional features;
(2) Multi-slice propagator, that propagates detection result to adjacent slice to reduce false negative;
(3) Motionless-guide suppressor based on the Random Forest algorithm, which builds features to describe motion, and reduces false positive with Random Forest.
Though tremendous progress has been made in object detection, one of the remaining major open challenges is to detect small objects such as 4x4 lung nodules.
The difficulty mainly results from two aspects:
(1) Features.
The high-level features of deep ConvNet integrate the low-level features to get rich semantic and context information, and realize high quality detection based on that information. However, small lung nodules may have little information in high-level features. In addition, low-level feature map has more information of small nodules than high-level, yet less context information to distinguish nodules from background. Also, since most image detection algorithms scan window in the feature map to generate bounding boxes and predict whether the object is present in those boxes, the high-level feature map has low resolution and it is difficult to generate sufficient bounding boxes which contain small nodules.
(2) Data imbalance.
Considering the lung nodule as positive samples, and other tissues as negative samples, the ratio of positive sample and negative sample is approximately 1:5000, which causes extreme data imbalance during training and make the model prediction prone to be negative. We explored some strategies to alleviate those problems when building the model.
When using single scale high-level feature, small object has little information in it.
When using the pyramidal feature hierarchy to predict multi-size objects, it is hard to get sufficient context and semantic information for small objects.
Propagating the high-level feature to low-level feature results in more context information and semantic information with high resolution.
We first clustered the 2D image detection results and organized them into 3D cubes, the we built a model to describe the motion information. With our model structure MSSD, we propagate high-level features from top to bottom to get features with both rich context information and semantic information.
Quality detection result and some typical false positives. Green and red rectangles respectively indicate true and false positives. Our model is insensitive to nodule characteristics including size, density and edge. False positives are quite similar in appearance to true positives.
We propose a novel approach to detect pulmonary lung nodule in CT images by treating this task as an VID task and applying some technologies such as still image detector, motion information analysis, which lead to promising result with high detection result. We analyze the similarity and difference between CT images and video frames, improve the still image detector to enable it to learn rich context information such as size and density as well as keeping rich information of small lung nodule.
We also propose two simple methods, MSP and MLSG, and utilize the motion information of lung nodules to reduce false negatives and false positives. We can see that the CNN, being a dominant trend in natural image processing, pervade quickly in the medical image analysis community.
However, there are still challenges when applying the deep learning technologies to medical image analysis:
1) some technologies that perform excellently on natural images may not work in medical image analysis. For example, we find that the residual model and inception module do not work as expected in detecting lung nodule in 2D images.
2) Labeled data with high quality are hard to acquire, which limits the deep learning usage in medical image analysis.
3) Training a reliable model from scratch is difficult, the prior knowledge of doctors is hard to integrate into the end-to-end deep learning structure.
The computation of deep ConvNet is intensive even with the high-end hardware and high-quality deep learning frame (e.g. MXNet), which limits its clinical usage. With our light-weighted method, diagnosis of a patient needs 20 seconds with a GTX1080. How to incorporate radiologists’ prior knowledge into deep ConvNet is still an open question.
